Method for analyzing biological specimens by spectral imaging

ABSTRACT

A method for registering a visual image and a spectral image of a biological sample includes aligning a first set of coordinate positions of a plurality of reticles on a slide holder and a second set of coordinate positions of the plurality of reticles on the slide holder. The method further includes generating a registered image of a visual image of a biological sample and a spectral image of the biological sample based upon the alignment of the first and second set of coordinate positions.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a Continuation of U.S. patent application Ser. No.13/507,386, filed Jun. 25, 2012, which is a Continuation-In-Part of U.S.patent application Ser. No. 13/067,777, filed on Jun. 24, 2011, whichclaims the benefit of U.S. Provisional Patent Application No.61/358,606, filed on Jun. 25, 2010. The entirety of each of theforegoing applications is hereby incorporated by reference herein.

FIELD OF THE INVENTION

Aspects of the invention relate to a method for analyzing biologicalspecimens by spectral imaging to provide a medical diagnosis, prognosticand/or predictive classification. The biological specimens may includemedical specimens obtained by surgical methods, biopsies, and culturedsamples.

BACKGROUND

Various pathological methods are used to analyze biological specimensfor the detection of abnormal or cancerous cells. For example, standardhistopathology involves visual analysis of stained tissue sections by apathologist using a microscope. Typically, tissue sections are removedfrom a patient by biopsy, and the samples are either snap frozen andsectioned using a cryo-microtome, or they are formalin-fixed, paraffinembedded, and sectioned via a microtome. The tissue sections are thenmounted onto a suitable substrate. Paraffin-embedded tissue sections aresubsequently deparaffinized. The tissue sections are stained using, forexample, an hemotoxylin-eosin (H&E) stain and are coverslipped.

The tissue samples are then visually inspected at a high resolutionvisual inspection, for example, 10× to 40× magnification. The magnifiedcells are compared with visual databases in the pathologist's memory.Visual analysis of a stained tissue section by a pathologist involvesscrutinizing features such as nuclear and cellular morphology, tissuearchitecture, staining patterns, and the infiltration of immune responsecells to detect the presence of abnormal or cancerous cells.

If early metastases or small clusters of cancerous cells measuring fromless than 0.2 to 2 mm in size, known as micrometastases, are suspected,adjacent tissue sections may be stained with an immuno-histochemical(IHC) agent/counter stain such as cytokeratin-specific stains. Suchmethods increase the sensitivity of histopathology since normal tissue,such as lymph node tissue, does not respond to these stains. Thus, thecontrast between unaffected and diseased tissue can be enhanced.

The primary method for detecting micrometastases has been standardhistopathology. The detection of micrometastases in lymph nodes, forexample, by standard histopathology is a formidable task owing to thesmall size and lack of distinguishing features of the abnormality withinthe tissue of a lymph node. Yet, the detection of these micrometastasesis of prime importance to stage the spread of disease because if a lymphnode is found to be free of metastatic cells, the spread of cancer maybe contained. On the other hand, a false negative diagnosis resultingfrom a missed micrometastasis in a lymph node presents too optimistic adiagnosis, and a more aggressive treatment should have been recommended.

Although standard histopathology is well-established for diagnosingadvanced diseases, it has numerous disadvantages. In particular,variations in the independent diagnoses of the same tissue section bydifferent pathologists are common because the diagnosis and grading ofdisease by this method is based on a comparison of the specimen ofinterest with a database in the pathologist's memory, which isinherently subjective. Differences in diagnoses particularly arise whendiagnosing rare cancers or in the very early stages of disease. Inaddition, standard histopathology is time consuming, costly and relieson the human eye for detection, which makes the results hard toreproduce. Further, operator fatigue and varied levels of expertise ofthe pathologist may impact a diagnosis.

In addition, if a tumor is poorly differentiated, manyimmunohistochemical stains may be required to help differentiate thecancer type. Such staining may be performed on multiple parallel cellblocks. This staining process may be prohibitively expensive andcellular samples may only provide a few diagnostic cells in a singlecell block.

To overcome the variability in diagnoses by standard histopathology,which relies primarily on cell morphology and tissue architecturalfeatures, spectroscopic methods have been used to capture a snapshot ofthe biochemical composition of cells and tissue. This makes it possibleto detect variations in the biochemical composition of a biologicalspecimen caused by a variety of conditions and diseases. By subjecting atissue or cellular sample to spectroscopy, variations in the chemicalcomposition in portions of the sample may be detected, which mayindicate the presence of abnormal or cancerous cells. The application ofspectroscopy to infrared cytopathology (the study of diseases of cells)is referred to as “spectral cytopathology” (SCP), and the application ofinfrared spectroscopy to histopathology (the study of diseases oftissue) as “spectral histopathology” (SHP).

SCP on individual urinary tract and cultured cells is discussed in B.Bird et al., Vibr. Spectrosc., 48, 10 (2008) and M. Romeo et al.,Biochim Biophys Acta, 1758, 915 (2006). SCP based on imaging data setsand applied to oral mucosa and cervical cells is discussed in WO2009/146425. Demonstration of disease progression via SCP in oralmucosal cells is discussed in K. Papamarkakis et al., LaboratoryInvestigations, 90, 589 (2010). Demonstration of sensitivity of SCP todetect cancer field effects and sensitivity to viral infection incervical cells is discussed in K. Papamarkakis et al., LaboratoryInvestigations, 90, 589, (2010).

Demonstration of first unsupervised imaging of tissue using SHP of livertissue via hierarchical cluster analysis (HCA) is discussed in M. Diemet al., Biopolymers, 57, 282 (2000). Detection of metastatic cancer inlymph nodes is discussed in M. J. Romeo et al., Vibrational Spectrosc.,38, 115 (2005) and M. Romeo et al., Vibrational Microspectroscopy ofCells and Tissues, Wiley-Interscience, Hoboken, N.J. (2008). Use ofneural networks, trained on HCA-derived data, to diagnose cancer incolon tissue is discussed in P. Lasch et al., J. Chemometrics, 20, 209(2007). Detection of micro-metastases and individual metastatic cancercells in lymph nodes is discussed in B. Bird et al., The Analyst, 134,1067 (2009), B. Bird et al., BMC J. Clin. Pathology, 8, 1 (2008), and B.Bird et al., Tech. Cancer Res. Treatment, 10, 135 (2011).

Spectroscopic methods are advantageous in that they alert a pathologistto slight changes in chemical composition in a biological sample, whichmay indicate an early stage of disease. In contrast, morphologicalchanges in tissue evident from standard histopathology take longer tomanifest, making early detection of disease more difficult.Additionally, spectroscopy allows a pathologist to review a largersample of tissue or cellular material in a shorter amount of time thanit would take the pathologist to visually inspect the same sample.Further, spectroscopy relies on instrument-based measurements that areobjective, digitally recorded and stored, reproducible, and amenable tomathematical/statistical analysis. Thus, results derived fromspectroscopic methods are more accurate and precise then those derivedfrom standard histopathological methods.

Various techniques may be used to obtain spectral data. For example,Raman spectroscopy, which assesses the molecular vibrations of a systemusing a scattering effect, may be used to analyze a cellular or tissuesample. This method is described in N. Stone et al., VibrationalSpectroscopy for Medical Diagnosis, J. Wiley & Sons (2008), and C.Krafft, et al., Vibrational Spectrosc. (2011).

Raman's scattering effect is considered to be weak in that only about 1in 10¹⁰ incident photons undergoes Raman scattering. Accordingly, Ramanspectroscopy works best using a tightly focused visible or near-IR laserbeam for excitation. This, in turn, dictates the spot from whichspectral information is being collected. This spot size may range fromabout 0.3 μm to 2 μm in size, depending on the numerical aperture of themicroscope objective, and the wavelength of the laser utilized. Thissmall spot size precludes data collection of large tissue sections,since a data set could contain millions of spectra and would requirelong data acquisition times. Thus, SHP using Raman spectroscopy requiresthe operator to select small areas of interest. This approach negatesthe advantages of spectral imaging, such as the unbiased analysis oflarge areas of tissue.

SHP using infrared spectroscopy has also been used to detectabnormalities in tissue, including, but not limited to brain, lung, oralmucosa, cervical mucosa, thyroid, colon, skin, breast, esophageal,prostate, and lymph nodes. Infrared spectroscopy, like Ramanspectroscopy, is based on molecular vibrations, but is an absorptioneffect, and between 1% and 50% of incident infrared photons are likelyto be absorbed if certain criteria are fulfilled. As a result, data canbe acquired by infrared spectroscopy more rapidly with excellentspectral quality compared to Raman spectroscopy. In addition, infraredspectroscopy is extremely sensitive in detecting small compositionalchanges in tissue. Thus, SHP using infrared spectroscopy is particularlyadvantageous in the diagnosis, treatment and prognosis of cancers suchas breast cancer, which frequently remains undetected until metastaseshave formed, because it can easily detect micro-metastases. It can alsodetect small clusters of metastatic cancer cells as small as a fewindividual cells. Further, the spatial resolution achievable usinginfrared spectroscopy is comparable to the size of a human cell, andcommercial instruments incorporating large infrared array detectors maycollect tens of thousands of pixel spectra in a few minutes.

A method of SHP using infrared spectroscopy is described in Bird et al.,“Spectral detection of micro-metastates in lymph node histo-pathology”,J. Biophoton. 2, No. 1-2, 37-46 (2009), (hereinafter “Bird”). Thismethod utilizes infrared micro-spectroscopy (IRMSP) and multivariateanalysis to pinpoint micro-metastases and individual metastatic cells inlymph nodes.

Bird studied raw hyperspectral imaging data sets including 25,600spectra, each containing 1650 spectral intensity points between 700 and4000 cm⁻¹. These data sets, occupying about 400 MByte each, wereimported and pre-processed. Data preprocessing included restriction ofthe wavenumber range to 900-1800 cm⁻¹ and other processes. The“fingerprint” infrared spectral region was further divided into a“protein region” between 1700 and 1450 cm⁻¹, which is dominated by theamide I and amide II vibrational bands of the peptide linkages ofproteins. This region is highly sensitive to different protein secondaryand tertiary structure and can be used to stage certain events in cellbiology that depend on the abundance of different proteins. The lowerwavenumber range, from 900 to 1350 cm⁻¹, the “phosphate region”,contains several vibrations of the phosphodiester linkage found inphospholipids, as well as DNA and RNA.

In Bird, a minimum intensity criterion for the integrated amide I bandwas imposed to eliminate pixels with no tissue coverage. Then, vectornormalization and conversion of the spectral vectors to secondderivatives was performed. Subsequently, data sets were subjectedindividually to hierarchical cluster analysis (HCA) using the Euclideandistance to define spectral similarity and Ward's algorithm forclustering. Pixel cluster membership was converted to pseudo-colorspectral images.

According to Bird's method, marks are placed on slides with a stainedtissue section to highlight areas that correspond to areas on theunstained adjacent tissue section that are to be subjected to spectralanalysis. The resulting spectral and visual images are matched by a userwho aligns specific features on the spectral image and the visual imageto physically overlay the spectral and visual images.

By Bird's method, corresponding sections of the spectral image and thevisual image are examined to determine any correlation between thevisual observations and the spectral data. In particular, abnormal orcancerous cells observed by a pathologist in the stained visual imagemay also be observed when examining a corresponding portion of thespectral image that overlays the stained visual image. Thus, theoutlines of the patterns in the pseudo-color spectral image maycorrespond to known abnormal or cancerous cells in the stained visualimage. Potentially abnormal or cancerous cells that were observed by apathologist in a stained visual image may be used to verify the accuracyof the pseudo-color spectral image.

Bird's method, however, is inexact because it relies on the skill of theuser to visually match specific marks on the spectral and visual images.This method is often imprecise. In addition, Bird's method allows thevisual and spectral images to be matched by physically overlaying them,but does not join the data from the two images to each other. Since theimages are merely physically overlaid, the superimposed images are notstored together for future analysis.

Further, since different adjacent sections of tissue are subjected tospectral and visual imaging, Bird's overlaid images do not display thesame tissue section. This makes it difficult to match the spectral andvisual images, since there may be differences in the morphology of thevisual image and the color patterns in the spectral image.

Another problem with Bird's overlaying method is that the visual imageis not in the same spatial domain as the infrared spectral image. Thus,the spatial resolution of Bird's visual image and spectral image aredifferent. Typically, spatial resolution in the infrared image is lessthan the resolution of the visual image. To account for this differencein resolution, the data used in the infrared domain may be expanded byselecting a region around the visual point of interest and diagnosingthe region, and not a single point. For every point in the visual image,there is a region in the infrared image that is greater than the pointthat must be input to achieve diagnostic output. This process ofaccounting for the resolution differences is not performed by Bird.Instead, Bird assumes that when selecting a point in the visual image,it is the same point of information in the spectral image through theoverlay, and accordingly a diagnostic match is reported. While theimages may visually be the same, they are not the same diagnostically.

To claim a diagnostic match, the spectral image used must be output froma supervised diagnostic algorithm that is trained to recognize thediagnostic signature of interest. Thus, the spectral image cluster willbe limited by the algorithm classification scheme to driven by abiochemical classification to create a diagnostic match, and not auser-selectable match. By contrast, Bird merely used an “unsupervised”HCA image to compare to a “supervised” stained visual image to make adiagnosis. The HCA image identifies regions of common spectral featuresthat have not yet been determined to be diagnostic, based on rules andlimits assigned for clustering, including manually cutting thedendrogram until a boundary (geometric) match is visually accepted bythe pathologist to outline a cancer region. This method merely providesa visual comparison.

Other methods based on the analysis of fluorescence data exist that aregenerally based on the distribution of an external tag, such as a stainor label, or utilize changes in the inherent fluorescence, also known asauto-fluorescence. These methods are generally less diagnostic, in termsof recognizing biochemical composition and changes in composition. Inaddition, these methods lack the fingerprint sensitivity of techniquesof vibrational spectroscopy, such as Raman and infrared.

A general problem with spectral acquisition techniques is that anenormous amount of spectral data is collected when testing a biologicalsample. As a result, the process of analyzing the data becomescomputationally complicated and time consuming. Spectral data oftencontains confounding spectral features that are frequently observed inmicroscopically acquired infrared spectra of cells and tissue, such asscattering and baseline artifacts. Thus, it is helpful to subject thespectral data to pre-processing to isolate the cellular material ofinterest, and to remove confounding spectral features.

One type of confounding spectral feature is Mie scattering, which is asample morphology-dependent effect. This effect interferes with infraredabsorption or reflection measurements if the sample is non-uniform andincludes particles the size of approximately the wavelength of the lightinterrogating the sample. Mie scattering is manifested by broad,undulating scattering features, onto which the infrared absorptionfeatures are superimposed.

Mie scattering may also mediate the mixing of absorptive and reflectiveline shapes. In principle, pure absorptive line shapes are thosecorresponding to the frequency-dependence of the absorptivity, and areusually Gaussian, Lorentzian or mixtures of both. The absorption curvescorrespond to the imaginary part of the complex refractive index.Reflective contributions correspond to the real part of the complexrefractive index, and are dispersive in line shapes. The dispersivecontributions may be obtained from absorptive line shapes by numericKK-transform, or as the real part of the complex Fourier transform (FT).

Resonance Mie (RMie) features result from the mixing of absorptive andreflective band shapes, which occurs because the refractive indexundergoes anomalous dispersion when the absorptivity goes through amaximum (i.e., over the profile of an absorption band). Mie scattering,or any other optical effect that depends on the refractive index, willmix the reflective and absorptive line shapes, causing a distortion ofthe band profile, and an apparent frequency shift.

FIG. 1 illustrates the contamination of absorption patterns bydispersive band shapes observed in both SCP and SHP. The bottom trace inFIG. 1 depicts a regular absorption spectrum of biological tissue,whereas the top trace shows a spectrum strongly contaminated by adispersive component via the RMie effect. The spectral distortionsappear independent of the chemical composition, but rather depend on themorphology of the sample. The resulting band intensity and frequencyshifts aggravate spectral analysis to the point that uncontaminated andcontaminated spectra are classified into different groups due to thepresence of the band shifts. Broad, undulating background features areshown in FIG. 2. When superimposed on the infrared micro-spectroscopy(IR-MSP) patterns of cells, these features are attributed to Miescattering by spherical particles, such as cellular nuclei, or sphericalcells.

The appearance of dispersive line shapes in FIG. 1 superimposed onIR-MSP spectra was reported along with a theoretical analysis in M.Romeo, et al., Vibrational Spectroscopy, 38, 129 (2005) (hereinafter“Romeo 2005”). Romeo 2005 indentifies the distorted band shapes asarising from the superposition of dispersive (reflective) componentsonto the absorption features of an infrared spectrum. These effects wereattributed to incorrect phase correction of the instrument controlsoftware. In particular, the acquired raw interferogram in FTIRspectroscopy frequently is “chirped” or asymmetric, and needs to besymmetrized before FT. This is accomplished by collecting a double sidedinterferogram over a shorter interferometer stroke, and calculating aphase correction to yield a symmetric interferogram.

In Romeo 2005, it was assumed that this procedure was not functioningproperly, which causes it to yield distorted spectral features. Anattempt was made to correct the distorted spectral features bycalculating the phase between the real and imaginary parts of thedistorted spectra, and reconstructing a power spectrum from the phasecorrected real and imaginary parts. Romeo 2005 also reported the factthat in each absorption band of an observed infrared spectrum, therefractive index undergoes anomalous dispersion. Under certaincircumstances, various amounts of the dispersive line shapes can besuperimposed, or mixed in, with the absorptive spectra.

The mathematical relationship between absorptive and reflective bandshapes is given by the Kramers-Kronig (KK) transformation, which relatesthe two physical phenomena. The mixing of dispersive (reflective) andabsorptive effects in the observed spectra was identified, and a methodto correct the effect via a procedure called “Phase Correction” (PC) isdiscussed in Romeo 2005. Although the cause of the mixing of dispersiveand absorptive contributions was erroneously attributed to instrumentsoftware malfunction, the principle of the confounding effect wasproperly identified. Due to the incomplete understanding of theunderlying physics, however, the proposed correction method did not workproperly.

P. Bassan et al., Analyst, 134, 1586 (2009) and P. Bassan et al.,Analyst, 134, 1171 (2009) demonstrated that dispersive and absorptiveeffects may mix via the “Resonance Mie Scattering” (RMieS) effect. Analgorithm and method to correct spectral distortion is described in P.Bassan et al., “Resonant Mie Scattering (RMieS) correction of infraredspectra from highly scattering biological samples”, Analyst, 135,268-277 (2010). This method is an extension of the “ExtendedMultiplicative Signal Correction” (EMSC) method reported in A. Kohler etal., Appl. Spectrosc., 59, 707 (2005) and A. Kohler et al., Appl.Spectrosc., 62, 259 (2008).

This method removes the non-resonant Mie scattering from infraredspectral datasets by including reflective components obtained viaKK-transform of pure absorption spectra into a multiple linearregression model. The method utilizes the raw dataset and a “reference”spectrum as inputs, where the reference spectrum is used both tocalculate the reflective contribution, and as a normalization feature inthe EMSC scaling. Since the reference spectrum is not known a priori,Bassan et al. use the mean spectrum of the entire dataset, or an“artificial” spectrum, such as the spectrum of a pure protein matrix, asa “seed” reference spectrum. After the first pass through the algorithm,each corrected spectrum may be used in an iterative approach to correctall spectra in the subsequent pass. Thus, a dataset of 1000 spectra willproduce 1000 RMieS-EMSC corrected spectra, each of which will be used asan independent new reference spectrum for the next pass, requiring1,000,000 correction runs. To carry out this algorithm, referred to asthe “RMieS-EMSC” algorithm, to a stable level of corrected outputspectra required a number of passes (˜10), and computation times thatare measured in days.

Since the RMieS-EMSC algorithm requires hours or days of computationtime, a fast, two-step method to perform the elimination of scatteringand dispersive line shapes from spectra was developed, as discussed inB. Bird, M. Miljković and M. Diem, “Two step resonant Mie scatteringcorrection of infrared micro-spectral data: human lymph node tissue”, J.Biophotonics, 3 (8-9) 597-608 (2010). This approach includes fittingmultiple dispersive components, obtained from KK-transform of pureabsorption spectra, as well as Mie scattering curves computed via thevan Hulst equation (see H. C. Van De Hulst, Light Scattering by SmallParticles, Dover, Mineola, N. Y., (1981)), to all the spectra in adataset via a procedure known as Extended Multiplicative SignalCorrection (EMSC) (see A. Kohler et al., Appl. Spectrosc., 62, 259(2008)) and reconstructing all spectra without these confoundingcomponents.

This algorithm avoids the iterative approach used in the RMieS-EMSCalgorithm by using uncontaminated reference spectra from the dataset.These uncontaminated reference spectra were found by carrying out apreliminary cluster analysis of the dataset and selecting the spectrawith the highest amide I frequencies in each cluster as the“uncontaminated” spectra. The spectra were converted to pure reflectivespectra via numeric KK transform and used as interference spectra, alongwith compressed Mie curves for RMieS correction as described above. Thisapproach is fast, but only works well for datasets containing a fewspectral classes.

In the case of spectral datasets containing many tissue types, however,the extraction of uncontaminated spectra can become tedious.Furthermore, under these conditions, it is unclear whether fitting allspectra in the dataset to the most appropriate interference spectrum isguaranteed. In addition, this algorithm requires reference spectra forcorrection, and works best with large datasets.

In light of the above, there remains a need for improved methods ofanalyzing biological specimens by spectral imaging to provide a medicaldiagnosis. Further, there is a need for an improved pre-processingmethod that is based on a revised phase correction approach, does notrequire input data, is computationally fast, and takes into account manytypes of confounding spectral contributions that are frequently observedin microscopically acquired infrared spectra of cells and tissue.

SUMMARY

One aspect of the invention relates to a method for analyzing biologicalspecimens by spectral imaging to provide a medical diagnosis. The methodincludes obtaining spectral and visual images of biological specimensand registering the images to detect abnormalities in the biologicalspecimen, such as, but not limited to, cell abnormalities, pre-cancerouscells, and cancerous cells. This method overcomes the obstaclesdiscussed above, among others, in that it eliminates the bias andunreliability of diagnoses and prognosis that are inherent in standardhistopathological, cytological, and other spectral methods.

Another aspect of the invention relates to a method for correctingconfounding spectral contributions that are frequently observed inmicroscopically acquired infrared spectra of cells and tissue byperforming a phase correction on the spectral data. This phasecorrection method may be used to correct various kinds of absorptionspectra that are contaminated by reflective components.

According to aspects of the invention, a method for analyzing biologicalspecimens by spectral imaging includes acquiring a spectral image of thebiological specimen, acquiring a visual image of the biologicalspecimen, and registering the visual image and spectral image.

A method of developing a data repository according to aspects of theinvention includes identifying a region of a visual image displaying adisease or condition, associating the region of the visual image tospectral data corresponding to the region, and storing the associationbetween the spectral data and the corresponding disease or condition.

A method of providing a medical diagnosis according to aspects of theinvention includes obtaining spectroscopic data for a biologicalspecimen, comparing the spectroscopic data for the biological specimento data in a repository that is associated with a disease or condition,determining any correlation between the repository data and thespectroscopic data for the biological specimen, and outputting adiagnosis associated with the determination.

A system for providing a medical diagnosis and prognosis, according toaspects of the invention includes a processor, a user interfacefunctioning via the processor, and a repository accessible by theprocessor, where spectroscopic data of a biological specimen isobtained, the spectroscopic data for the biological specimen is comparedto repository data that is associated with a disease or condition, anycorrelation between the repository data and the spectroscopic data forthe biological specimen is determined; and a diagnosis associated withthe determination is output.

A computer program product according to aspects of the inventionincludes a computer usable medium having control logic stored thereinfor causing a computer to provide a medical diagnosis. The control logicincludes a first computer readable program code means for obtainingspectroscopic data for a biological specimen, second computer readableprogram code means for comparing the spectroscopic data for thebiological specimen to repository data that is associated with a diseaseor condition, third computer readable program code means for determiningany correlation between the repository data and the spectroscopic datafor the biological specimen, and fourth computer readable program codemeans for outputting a diagnosis and/or or a prognostic determinationassociated with the determination.

DESCRIPTION OF THE DRAWINGS

The file of this patent contains at least one drawing executed in color.Copies of this patent with color drawing(s) will be provided by thePatent and Trademark Office upon request and payment of the necessaryfee.

FIG. 1 illustrates the contamination of absorption patterns bydispersive band shapes typically observed in both SCP and SHP;

FIG. 2 shows broad, undulating background features typically observed onIR-MSP spectral of cells attributed to Mie scattering by sphericalparticles;

FIG. 3 is a flowchart illustrating a method of analyzing a biologicalsample by spectral imaging according to aspects of the invention;

FIG. 3A is a flowchart illustrating steps in a method of acquiring aspectral image according to aspects of the invention;

FIG. 3B is a flowchart illustrating steps in a method of pre-processingspectral data according to aspects of the invention;

FIGS. 4A and 4B are a flowchart illustrating an example method ofperforming image registration on a spectral image and a visual image inaccordance with aspects of the present invention;

FIG. 4C illustrates an example slide holder in accordance with aspectsof the present invention;

FIG. 5A is a flowchart illustrating an example method of refining imageregistration in accordance with aspects of the present invention;

FIG. 5B is an example a graphical user interface (GUI) interface forsetting a threshold value in accordance with aspects of the presentinvention;

FIG. 5C is an example GUI interface illustrating an example optimizationwindow in accordance with aspects of the present invention;

FIG. 6A shows a typical spectrum, superimposed on a linear backgroundaccording to aspects of the invention;

FIG. 6B shows an example of a second derivative spectrum according toaspects of the invention;

FIG. 7 shows a portion of the real part of an interferogram according toaspects of the invention;

FIG. 8 shows that the phase angle that produces the largest intensityafter phase correction is assumed to be the uncorrupted spectrumaccording to aspects of the invention;

FIG. 9A shows that absorption spectra that are contaminated byscattering effects that mimic a baseline slope according to aspects ofthe invention;

FIG. 9B shows that the imaginary part of the forward FT exhibitsstrongly curved effects at the spectral boundaries, which willcontaminate the resulting corrected spectra according to aspects of theinvention;

FIG. 10A is H&E-based histopathology showing a lymph node that hasconfirmed breast cancer micro-metastases under the capsule according toaspects of the invention;

FIG. 10B shows data segmentation by Hierarchical Cluster Analysis (HCA)carried out on the lymph node section of FIG. 10A according to aspectsof the invention;

FIG. 10C is a plot showing the peak frequencies of the amide Ivibrational band in each spectrum according to aspects of the invention;

FIG. 10D shows an image of the same lymph node section of FIG. 10A afterphase-correction using RMieS correction according to aspects of theinvention;

FIG. 11A shows the results of HCA after phase-correction using RMieScorrection of FIG. 10D according to aspects of the invention;

FIG. 11B is H&E-based histopathology of the lymph node section of FIG.11A according to aspects of the invention;

FIG. 12A is a visual microscopic image of a section of stained cervicalimage;

FIG. 12B is an infrared spectral image created from hierarchical clusteranalysis of an infrared dataset collected prior to staining the tissueaccording to aspects of the invention;

FIG. 13A is a visual microscopic image of a section of an H&E-stainedaxillary lymph node section according to aspects of the invention;

FIG. 13B is an infrared spectral image created from a MultilayerPerceptron Networks analysis of an infrared dataset collected prior tostaining the tissue according to aspects of the invention;

FIG. 14A is a visual image of a small cell lung cancer tissue accordingto aspects of the invention.

FIG. 14B is an HCA-based spectral image of the tissue shown in FIG. 14Aaccording to aspects of the invention;

FIG. 14C is a registered image of the visual image of FIG. 14A and thespectral image of FIG. 14B, according to aspects of the invention;

FIG. 14D is an example of a graphical user interface (GUI) for theregistered image of FIG. 14C according to aspects of the invention;

FIG. 15A is a visual microscopic image of H&E-stained lymph node tissuesection according to aspects of the invention;

FIG. 15B is a global digital staining image of section shown in FIG.15A, distinguishing capsule and interior of lymph node according toaspects of the invention;

FIG. 15C is a diagnostic digital staining image of the section shown inFIG. 15A, distinguishing capsule, metastatic breast cancer, histiocytes,activated B-lymphocytes, and T-lymphocytes according to aspects of theinvention;

FIG. 16 is a schematic of relationship between global and diagnosticdigital staining according to aspects of the invention;

FIG. 17A is a visual image of H&E-stained tissue section from anaxillary lymph node according to aspects of the invention;

FIG. 17B is a SHP-based digitally stained region of breast cancermicrometastasis according to aspects of the invention;

FIG. 17C is a SHP-based digitally stained region occupied byB-lymphocyes according to aspects of the invention;

FIG. 17D is a SHP-based digitally stained region occupied by histocytesaccording to aspects of the invention.

FIG. 18 illustrates the detection of individual cancer cells, and smallclusters of cancer cells via SHP according to aspects of the invention;

FIG. 19A shows raw spectral data sets comprising cellular spectrarecorded from lung adenocarcinoma, small cell carcinoma, and squamouscell carcinoma cells according to aspects of the invention;

FIG. 19B shows corrected spectral data sets comprising cellular spectrarecorded from lung adenocarcinoma, small cell carcinoma, and squamouscell carcinoma cells according to aspects of the invention;

FIG. 19C shows standard spectra for lung adenocarcinoma, small cellcarcinoma, and squamous cell carcinoma according to aspects of theinvention;

FIG. 19D shows KK transformed spectra calculated from spectra in FIG.19C;

FIG. 19E shows PCA scores plots of the multi class data set before EMSCcorrection according to aspects of the invention;

FIG. 19F shows PCA scores plots of the multi class data set after EMSCcorrection according to aspects of the invention;

FIG. 20A shows mean absorbance spectra of lung adenocarcinoma, smallcell carcinoma, and squamous carcinoma, according to aspects of theinvention;

FIG. 20B shows second derivative spectra of absorbance spectra displayedin FIG. 20A according to aspects of the invention;

FIG. 21A shows 4 stitched microscopic R&E-stained images of 1 mm×1 mmtissue areas comprising adenocarcinoma, small cell carcinoma, andsquamous cell carcinoma cells, respectively, according to aspects of theinvention;

FIG. 21B is a binary mask image constructed by performance of a rapidreduced RCA analysis upon the 1350 cm⁻¹-900 cm⁻¹ spectral region of the4 stitched raw infrared images recorded from the tissue areas shown inFIG. 21A according to aspects of the invention;

FIG. 21C is a 6-cluster RCA image of the scatter corrected spectral datarecorded from regions of diagnostic cellular material according toaspects of the invention;

FIG. 22 shows various features of a computer system for use inconjunction with aspects of the invention; and

FIG. 23 shows a computer system for use in conjunction with aspects ofthe invention;

DETAILED DESCRIPTION

Unless otherwise defined, all technical and scientific terms used hereinhave the same meaning as commonly understood by one of ordinary skill inthe art to which aspects of this invention belong. Although methods andmaterials similar or equivalent to those described herein can be used inthe practice or testing, suitable methods and materials are describedbelow. All publications, patent applications, patents, and otherreferences mentioned herein are incorporated by reference in theirentirety. In case of conflict, this specification, includingdefinitions, will control. In addition, the materials, methods, andexamples are illustrative only and not intended to be limiting.

One aspect of the invention relates to a method for analyzing biologicalspecimens by spectral imaging to provide a medical diagnosis. Thebiological specimens may be medical specimens obtained by surgicalmethods, biopsies, and cultured samples. The method includes obtainingspectral and visual images of biological specimens and registering theimages to detect cell abnormalities, pre-cancerous cells, and cancerouscells. The biological specimens may include tissue or cellular samples,but tissue samples are preferred for some applications. This methodidentifies abnormal or cancerous and other disorders including, but notlimited to, lymph node, thyroid, breast, uterine, renal, testicular,ovarian, or prostate cancer, small cell lung carcinoma, non-small celllung carcinoma, and melanoma, as well as non-cancerous effectsincluding, but not limited to, inflammation, necrosis, and apoptosis.

One method in accordance with aspects of the invention overcomes theobstacles discussed above in that it eliminates or generally reduces thebias and unreliability of diagnoses, prognosis, predictive, andtheranostics that are inherent in standard histopathological and otherspectral methods. In addition, it allows access to a spectral databaseof tissue types that is produced by quantitative and reproduciblemeasurements and is analyzed by an algorithm that is calibrated againstclassical histopathology. Via this method, for example, abnormal andcancerous cells may be detected earlier than they can be identified bythe related art, including standard histopathological or other spectraltechniques.

A method in accordance with aspects of the invention is illustrated inthe flowchart of FIG. 3. As shown in FIG. 3, the method generallyincludes the steps of acquiring a biological section 301, acquiring aspectral image of the biological section 302, acquiring a visual imageof the same biological section 303, and performing image registration304. The registered image may optionally be subjected to training 305,and a medical diagnosis may be obtained 306.

Biological Section

According to the example method of the invention shown in FIG. 3, thestep of acquiring a biological section 301 refers to the extraction oftissue or cellular material from an individual, such as a human oranimal. A tissue section may be obtained by methods including, but notlimited to core and punch biopsy, and excising. Cellular material may beobtained by methods including, but not limited to swabbing(exfoliation), washing (lavages), and by fine needle aspiration (FNA).

A tissue section that is to be subjected to spectral and visual imageacquisition may be prepared from frozen or from paraffin embedded tissueblocks according to methods used in standard histopathology. The sectionmay be mounted on a slide that may be used both for spectral dataacquisition and visual pathology. For example, the tissue may be mountedeither on infrared transparent microscope slides comprising a materialincluding, but not limited to, calcium fluoride (CaF₂) or on infraredreflective slides, such as commercially available “low-e” slides. Aftermounting, paraffin-embedded samples may be subjected todeparaffinization.

Spectral Image

According to aspects of the invention, the step of acquiring a spectralimage of the biological section 302 shown in FIG. 3 may includeacquiring spectral data from the biological section 308, performing datapre-processing 310, performing multivariate analysis 312, and creating agrayscale or pseudo-color image of the biological section 314, asoutlined in the flowchart of FIG. 3A.

Spectral Data

As set forth in FIG. 3A, spectral data from the biological section maybe acquired 401. Spectral data from an unstained biological sample, suchas a tissue sample, may be obtained to capture a snapshot of thechemical composition of the sample. The spectral data may be collectedfrom a tissue section in pixel detail, where each pixel is about thesize of a cellular nucleus. Each pixel has its own spectral pattern, andwhen the spectral patterns from a sample are compared, they may showsmall but recurring differences in the tissue's biochemical composition.

The spectral data may be collected by methods including, but not limitedto infrared, Raman, visible, terahertz, and fluorescence spectroscopy.Infrared spectroscopy may include, but is not limited to, attenuatedtotal reflectance (ATR) and attenuated total reflectance Fouriertransform infrared spectroscopy (ATR-FTIR). In general, infraredspectroscopy may be used because of its fingerprint sensitivity, whichis also exhibited by Raman spectroscopy. Infrared spectroscopy may beused with larger tissue sections and to provide a dataset with a moremanageable size than Raman spectroscopy. Furthermore, infraredspectroscopy data may be more amenable to fully automatic dataacquisition and interpretation. Additionally, infrared spectroscopy mayhave the necessary sensitivity and specificity for the detection ofvarious tissue structures and diagnosis of disease.

The intensity axis of the spectral data, in general, express absorbance,reflectance, emittance, scattering intensity or any other suitablemeasure of light power. The wavelength may relate to the actualwavelength, wavenumber, frequency or energy of electromagneticradiation.

Infrared data acquisition may be carried out using presently availableFourier transform (FT) infrared imaging microspectrometers, tunablelaser-based imaging instruments, such as quantum cascade or non-linearoptical devices, or other functionally equivalent instruments based ondifferent technologies. The acquisition of spectral data using a tunablelaser is described further in U.S. patent application Ser. No.13/084,287 titled, “Tunable Laser-Based Infrared Imaging System andMethod of Use Thereof”, which is incorporated herein in its entirety byreference.

According to one method in accordance with aspects of the invention, apathologist or technician may select any region of a stained tissuesection and receive a spectroscopy-based assessment of the tissue regionin real-time, based on the hyperspectral dataset collected for thetissue before staining. Spectral data may be collected for each of thepixels in a selected unstained tissue sample. Each of the collectedspectra contains a fingerprint of the chemical composition of each ofthe tissue pixels. Acquisition of spectral data is described in WO2009/146425, which is incorporated herein in its entirety by reference.

In general, the spectral data includes hyperspectral datasets, which areconstructs including N=n·m individual spectra or spectral vectors(absorption, emission, reflectance etc.), where n and m are the numberof pixels in the x and y dimensions of the image, respectively. Eachspectrum is associated with a distinct pixel of the sample, and can belocated by its coordinates x and y, where 1≦x≦n, and 1≦y≦m. Each vectorhas k intensity data points, which are usually equally spaced in thefrequency or wavenumber domain.

The pixel size of the spectral image may generally be selected to besmaller than the size of a typical cell so that subcellular resolutionmay be obtained. The size may also be determined by the diffractionlimit of the light, which is typically about 5 μm to about 7 μm forinfrared light. Thus, for a 1 mm² section of tissue, about 140² to about200² individual pixel infrared spectra may be collected. For each of theN pixels of a spectral “hypercube”, its x and y coordinates and itsintensity vector (intensity vs. wavelength), are stored.

Pre-Processing

Subjecting the spectral data to a form of pre-processing may be helpfulto isolate the data pertaining to the cellular material of interest andto remove confounding spectral features. Referring to FIG. 3A, once thespectral data is collected, it may be subjected to such pre-processing310.

Pre-processing may involve creating a binary mask to separate diagnosticfrom non-diagnostic regions of the sampled area to isolate the cellulardata of interest. Methods for creating a binary mask are disclosed in WO2009/146425, which is incorporated by reference herein in its entirety.

A method of pre-processing, according to another aspect of theinvention, permits the correction of dispersive line shapes in observedabsorption spectra by a “phase correction” algorithm that optimizes theseparation of real and imaginary parts of the spectrum by adjusting thephase angle between them. This method, which is computationally fast, isbased on a revised phase correction approach, in which no input data arerequired. Although phase correction is used in the pre-processing of rawinterferograms in FTIR and NMR spectroscopy (in the latter case, theinterferogram is usually referred to as the “free induction decay, FID”)where the proper phase angle can be determined experimentally, themethod of this aspect of the invention differs from earlier phasecorrection approaches in that it takes into account mitigating factors,such as Mie, RMie and other effects based on the anomalous dispersion ofthe refractive index, and it may be applied to spectral datasetsretroactively.

The pre-processing method of this aspect of the invention transformscorrupted spectra into Fourier space by reverse FT transform. Thereverse FT results in a real and an imaginary interferogram. The secondhalf of each interferogram is zero-filled and forward FT transformedindividually. This process yields a real spectral part that exhibits thesame dispersive band shapes obtained via numeric KK transform, and animaginary part that includes the absorptive line shapes. By recombiningthe real and imaginary parts with a correct phase angle between them,phase-corrected, artifact-free spectra are obtained.

Since the phase required to correct the contaminated spectra cannot bedetermined experimentally and varies from spectrum to spectrum, phaseangles are determined using a stepwise approach between −90° and 90° inuser selectable steps. The “best” spectrum is determined by analysis ofpeak position and intensity criteria, both of which vary during phasecorrection. The broad undulating Mie scattering contributions are notexplicitly corrected for explicitly in this approach, but they disappearby performing the phase correction computation on second derivativespectra, which exhibit a scatter-free background.

According to aspects of the invention, pre-processing 310 as shown inFIG. 3A may include selecting the spectral range 316, computing thesecond derivative of the spectra 318, reverse Fourier transforming thedata 320, zero-filling and forward Fourier transforming theinterferograms 322, and phase correcting the resulting real andimaginary parts of the spectrum 324, as outlined in the flowchart ofFIG. 3B.

Spectral Range

In 316, each spectrum in the hyperspectral dataset is pre-processed toselect the most appropriate spectral range (fingerprint region). Thisrange may be about 800 to about 1800 cm⁻¹, for example, which includesheavy atom stretching as well as X-H (X: heavy atom with atomic number≦12) deformation modes. A typical example spectrum, superimposed on alinear background, is shown in FIG. 6A.

Second Derivative of Spectra

The second derivative of each spectrum is then computed 318 as shown inthe flowchart of FIG. 3B. Second derivative spectra are derived fromoriginal spectral vectors by second differentiation of intensity vs.wavenumber. Second derivative spectra may be computed using aSavitzky-Golay sliding window algorithm, and can also be computed inFourier space by multiplying the interferogram by an appropriatelytruncated quadratic function.

Second derivative spectra may have the advantage of being free ofbaseline slopes, including the slowly changing Mie scatteringbackground. The second derivative spectra may be nearly completelydevoid of baseline effects due to scattering and non-resonant Miescattering, but still contain the effects of RMieS. The secondderivative spectra may be vector normalized, if desired, to compensatefor varying sample thickness. An example of a second derivative spectrumis shown in FIG. 6B.

Reverse Fourier Transform

As shown in 320 of the flowchart of FIG. 3B, each spectrum of the dataset is reverse Fourier transformed (FT). Reverse FT refers to theconversion of a spectrum from intensity vs. wavenumber domain tointensity vs. phase difference domain. Since FT routines only work withspectral vectors the length of which are an integer power of 2, spectraare interpolated or truncated to 512, 1024 or 2048 (NFT) data pointlength before FT. Reverse FT yields a real (RE) and imaginary (IM)interferogram of NFT/2 points. A portion of the real part of such aninterferogram is shown in FIG. 7.

Zero-Fill and Forward Fourier Transform

The second half of both the real and imaginary interferogram for eachspectrum is subsequently zero-filled 322. These zero-filledinterferograms are subsequently forward Fourier transformed to yield areal and an imaginary spectral component with dispersive and absorptiveband shapes, respectively.

Phase Correction

The real (RE) and imaginary (IM) parts resulting from the Fourieranalysis are subsequently phase corrected 324, as shown in the flowchartof FIG. 3B. This yields phase shifted real (RE′) and imaginary (IM′)parts as set forth in the formula below:

${\begin{pmatrix}{RE}^{\prime} \\{IM}^{\prime}\end{pmatrix}\begin{matrix} = \\ = \end{matrix}\begin{pmatrix}{\cos (\varphi)} & {\sin (\varphi)} \\{- {\sin (\varphi)}} & {\cos (\varphi)}\end{pmatrix}\begin{pmatrix}{RE} \\{IM}\end{pmatrix}},$

where φ is the phase angle.

Since the phase angle φ for the phase correction is not known, the phaseangle may be varied between −π/2≦φ≦π/2 in user defined increments, and aspectrum with the least residual dispersive line shape may be selected.The phase angle that produces the largest intensity after phasecorrection may be assumed to be the uncorrupted spectrum, as shown inFIG. 8. The heavy trace marked with the arrows and referred to as the“original spectrum” is a spectrum that is contaminated by RMieScontributions. The thin traces show how the spectrum changes upon phasecorrection with various phase angles. The second heavy trace is therecovered spectrum, which matches the uncontaminated spectrum well. Asindicated in FIG. 8, the best corrected spectrum exhibits the highestamide I intensity at about 1655 cm⁻¹. This peak position matches theposition before the spectrum was contaminated.

The phase correction method, in accordance with aspects of the inventionin 316-324, works well both with absorption and derivative spectra. Thisapproach even solves a complication that may occur if absorption spectraare used, in that if absorption spectra are contaminated by scatteringeffects that mimic a baseline slope, as shown schematically in FIG. 9A,the imaginary part of the forward FT exhibits strongly curved effects atthe spectral boundaries, as shown in FIG. 9B, which will contaminate theresulting corrected spectra. Use of second derivative spectra mayeliminate this effect, since the derivation eliminates the slopingbackground; thus, artifact-free spectra may be obtained. Since theensuing analysis of the spectral data-set by hierarchical clusteranalysis, or other appropriate segmenting or diagnostic algorithms, iscarried out on second derivative spectra anyway, it is advantageous tocarry out the dispersive correction on second derivative spectra, aswell. Second derivative spectra exhibit reversal of the sign of spectralpeaks. Thus, the phase angle is sought that causes the largest negativeintensity. The value of this approach may be demonstrated fromartificially contaminated spectra: since a contamination with areflective component will always decrease its intensity, theuncontaminated or “corrected” spectrum will be the one with the largest(negative) band intensity in the amide I band between 1650 and 1660cm⁻¹.

Example 1 Operation of Phase Correction Algorithm

An example of the operation of the phase correction algorithm isprovided in FIGS. 10 and 11. This example is based on a datasetcollected from a human lymph node tissue section. The lymph node hasconfirmed breast cancer micro-metastases under the capsule, shown by theblack arrows in FIG. 10A. This photo-micrograph shows distinct cellularnuclei in the cancerous region, as well as high cellularity in areas ofactivated lymphocytes, shown by the gray arrow. Both these sampleheterogeneities contribute to large RMieS effects.

When data segmentation by hierarchical cluster analysis (HCA) was firstcarried out on this example lymph node section, the image shown in FIG.10B was obtained. To distinguish the cancerous tissue (dark green andyellow) from the capsule (red), and the lymphocytes (remainder ofcolors), 10 clusters were necessary, and the distinction of these tissuetypes was poor. In FIG. 10B, the capsule shown in red includes more thanone spectral class, which were combined into 1 cluster.

The difficulties in segmenting this dataset can be gauged by inspectionof FIG. 10C. This plot depicts the peak frequencies of the amide Ivibrational band in each spectrum. The color scale at right of thefigure indicates that the peak occurs between about 1630 and 1665 cm⁻¹of the lymph node body, and between 1635 and 1665 cm⁻¹ for the capsule.The spread of amide I frequency is typical for a dataset heavilycontaminated by RMieS effects, since it is well-known that the amide Ifrequency for peptides and proteins should occur in the range from 1650to 1660 cm⁻¹, depending on the secondary protein structure. FIG. 10Dshows an image of the same tissue section after phase-correction basedRMieS correction. Within the body of the lymph node, the frequencyvariation of the amide I peak was reduced to the range of 1650 to 1654cm⁻¹, and for the capsule to a range of 1657 to 1665 cm⁻¹(fibro-connective proteins of the capsule are known to consist mostly ofcollagen, a protein known to exhibit a high amide I band position).

The results from a subsequent HCA are shown in FIG. 11. In FIG. 11A,cancerous tissue is shown in red; the outline of the cancerous regionscoincides well with the H&E-based histopathology shown in FIG. 11B (thisfigure is the same as 10A). The capsule is represented by two differenttissue classes (light blue and purple), with activated B-lymphocytesshown in light green. Histiocytes and T-lymphocytes are shown in darkgreen, gray and blue regions. The regions depicted in FIG. 11A match thevisual histopathology well, and indicate that the phase correctionmethod discussed herein improved the quality of the spectralhistopathology methods enormously. In an aspect, narrow bandnormalization may also be used to enhance and/or improve the quality ofthe image, which may be helpful for image registration accuracy. Thenarrow band normalization may select features and/or subsets of featureswithin the broad band spectral region and apply a weighting to theselected features.

The advantages of the pre-processing method in accordance with aspectsof the invention over previous methods of spectral correction includethat the method provides a fast execution time of about 5000spectra/second, and no a priori information on the dataset is required.In addition, the phase correction algorithm can be incorporated intospectral imaging and “digital staining” diagnostic routines forautomatic cancer detection and diagnosis in SCP and SHP. Further, phasecorrection greatly improves the quality of the image, which is helpfulfor image registration accuracy and in diagnostic alignment and boundaryrepresentations.

Further, the pre-processing method in accordance with aspects of theinvention may be used to correct a wide range of absorption spectracontaminated by reflective components. Such contamination occursfrequently in other types of spectroscopy, such as those in which bandshapes are distorted by dispersive line shapes, such as DiffuseReflectance Fourier Transform Spectroscopy (DRIFTS), Attenuated TotalReflection (ATR), and other forms of spectroscopy in which mixing of thereal and imaginary part of the complex refractive index, or dielectricsusceptibility, occurs to a significant extent, such as may be presentwith Coherent Anti-Stokes Raman Spectroscopy (CARS).

Multivariate Analysis

Multivariate analysis may be performed on the pre-processed spectraldata to detect spectral differences, as outlined in 312 of the flowchartof FIG. 3A. In certain multivariate analyses, spectra are groupedtogether based on similarity. The number of groups may be selected basedon the level of differentiation required for the given biologicalsample. In general, the larger the number of groups, the more detailthat will be evident in the spectral image. A smaller number of groupsmay be used if less detail is desired. According to aspects of theinvention, a user may adjust the number of groups to attain the desiredlevel of spectral differentiation.

For example, unsupervised methods, such as HCA and principal componentanalysis (PCA), supervised methods, such as machine learning algorithmsincluding, but not limited to, artificial neural networks (ANNs),hierarchical artificial neural networks (hANN), support vector machines(SVM), and/or “random forest” algorithms may be used. Unsupervisedmethods are based on the similarity or variance in the dataset,respectively, and segment or cluster a dataset by these criteria,requiring no information except the dataset for the segmentation orclustering. Thus, these unsupervised methods create images that arebased on the natural similarity or dissimilarity (variance) in thedataset. Supervised algorithms, on the other hand, require referencespectra, such as representative spectra of cancer, muscle, or bone, forexample, and classify a dataset based on certain similarity criteria tothese reference spectra.

HCA techniques are disclosed in Bird (Bird et al., “Spectral detectionof micro-metastates in lymph node histo-pathology”, J. Biophoton. 2, No.1-2, 37-46 (2009)), which is incorporated herein in its entirety. PCA isdisclosed in WO 2009/146425, which is incorporated by reference hereinin its entirety.

Examples of supervised methods for use in accordance with aspects of theinvention may be found in P. Lasch et al. “Artificial neural networks assupervised techniques for FT-IR microspectroscopic imaging” J.Chemometrics 2006 (hereinafter “Lasch”); 20: 209-220, M. Miljkovic etal., “Label-free imaging of human cells: algorithms for imagereconstruction of Raman hyperspectral datasets” (hereinafter“Miljkovic”), Analyst, 2010, xx, 1-13, and A. Dupuy et al., “CriticalReview of Published Microarray Studies for Cancer Outcome and Guidelineson Statistical Analysis and Reporting”, JNCI, Vol. 99, Issue 2|Jan. 17,2007 (hereinafter “Dupuy”), each of which is incorporated by referenceherein in its entirety.

Grayscale or Pseudo-Color Spectral Image

Similarly grouped data from the multivariate analysis may be assignedthe same color code. The grouped data may be used to construct“digitally stained” grayscale or pseudo-color maps, as set forth in 314of the flowchart of FIG. 3A. Accordingly, this method may provide animage of a biological sample that is based solely or primarily on thechemical information contained in the spectral data.

An example of a spectral image prepared after multivariate analysis byHCA is provided in FIGS. 12A and 12B. FIG. 12A is a visual microscopicimage of a section of stained cervical image, measuring about 0.5 mm×1mm. Typical layers of squamous epithelium are indicated. FIG. 12B is apseudo-color infrared spectral image constructed after multivariateanalysis by HCA prior to staining the tissue. This image was created bymathematically correlating spectra in the dataset with each other, andis based solely on spectral similarities; no reference spectra wereprovided to the computer algorithm. As shown in FIG. 12B, an HCAspectral image may reproduce the tissue architecture visible aftersuitable staining (for example, with a H&E stain) using standardmicroscopy, as shown in FIG. 12A. In addition, FIG. 12B shows featuresthat are not readily detected in FIG. 12A, including deposits of keratinat (a) and infiltration by immune cells at (b).

The construction of pseudo-color spectral images by HCA analysis isdiscussed in Bird.

An example of a spectral image prepared after analysis by ANN isprovided in FIGS. 13A and 13B. FIG. 13A is a visual microscopic image ofa section of an H&E-stained axillary lymph node section. FIG. 13B is aninfrared spectral image created from ANN analysis of an infrared datasetcollected prior to staining the tissue of FIG. 13A.

Visual Image

A visual image of the same biological section obtained in 302 may beacquired, as indicated by 303 as shown in FIG. 3. The biological sampleapplied to a slide in step 301 described above may be unstained or maybe stained by any suitable well-known method used in standardhistopathology, such as by one or more H&E and/or IHC stains, and may becoverslipped. Examples of visual images are shown in FIGS. 12A and 13A.

A visual image of a histopathological sample may be obtained using astandard visual microscope, such as one commonly used in pathologylaboratories. The microscope may be coupled to a high resolution digitalcamera that captures the field of view of the microscope digitally. Thisdigital real-time image is based on the standard microscopic view of astained piece of tissue, and is indicative of tissue architecture, cellmorphology and staining patterns. The digital image may include manypixel tiles that are combined via image stitching, for example, tocreate a photograph. According to aspects of the invention, the digitalimage that is used for analysis may include an individual tile or manytiles that are stitched combined into a photograph. This digital imagemay be saved and displayed on a computer screen.

Registration of Spectral and Visual Images

According to one method in accordance with aspects of the invention,once the spectral and visual images have been acquired, the visual imageof the stained tissue may be registered with a digitally stainedgrayscale or pseudo-color spectral image, as indicated in 304 of theflowchart of FIG. 3. In general, image registration is the process oftransforming or matching different sets of data into one coordinatesystem. Image registration involves spatially matching or transforming afirst image to align with a second image. The pixels in the first imageand the pixels in the second image may coincide to the same points inthe coordinate system. The images may contain different types of data,and image registration allows the matching or transformation of thedifferent types of data. In an aspect, the transformation may include ascaled rigid body transformation. It should be noted that thetransformation may include warping if staining the sample made thesample shrink non-uniformly. Example transformation equations that thecomputing system may use include the following:

u=u0+scale*(x*cos(θ)−y*sin(θ))

v=v0+scale*(x*sin(θ)+y*cos(θ))

where (u0,v0) is a shift of the origin, θ is a rotation angle in radiansand scale is the scale factor, (x,y) are coordinates in the HCA image,and (u,v) are coordinates in the H&E (visual image).

In accordance with aspects of the invention, image registration may beperformed in a number of ways. For example, a common coordinate systemmay be established for the visual and spectral images. If establishing acommon coordinate system is not possible or is not desired, the imagesmay be registered by point mapping to bring an image into alignment withanother image. In point mapping, control points on both of the imagesthat identify the same feature or landmark in the images are selected.Based on the positions of the control points, spatial mapping of bothimages may be performed. For example, at least two control points may beused. To register the images, the control points in the visible imagemay be correlated to the corresponding control points in the spectralimage and aligned together.

In an aspect, at least two control points may be used to determine thetransformation parameters of the scaled body transformation. Thetransformation parameters may be selected to minimize an error betweenthe mapped control points in the registered images (e.g., the overlappedimages). For example, when two control points are used to determine thetransformation parameters, two solutions for the transformation may begenerated by the computing system. The computing system may select oneof the two solutions generated based upon, for example, the orientationof the image. However, when three control points are used to determinethe transformation parameters, a unique solution for the transformationmay be generated by the computing system. Thus, it should be noted thatmore than two control points may be used by the computing system todetermine the parameters of the scaled body transformation. In addition,as the number of control points increase, the accuracy of thetransformation may also increase and/or improve.

In one variation according to aspects of the invention, control pointsmay be selected by placing reference marks on the slide containing thebiological specimen. Reference marks may include, but are not limitedto, ink, paint, and a piece of a material, including, but not limited topolyethylene. The reference marks may have any suitable shape or size,and may be placed in the central portion, edges, or corners of the side,as long as they are within the field of view. The reference mark may beadded to the slide while the biological specimen is being prepared. If amaterial having known spectral patterns, including, but not limited to achemical substance, such as polyethylene, and a biological substance, isused in a reference mark, it may be also used as a calibration mark toverify the accuracy of the spectral data of the biological specimen.

In another variation according to aspects of the invention, a user, suchas a pathologist, may select the control points in the spectral andvisual images. The user may select the control points based on theirknowledge of distinguishing features of the visual or spectral imagesincluding, but not limited to, edges and boundaries. For biologicalimages such as cells and tissue, control points may be selected from anyof the biological features in the image. For example, such biologicalfeatures may include, but are not limited to, clumps of cells, mitoticfeatures, cords or nests of cells, sample voids, such as alveolar andbronchi, and irregular sample edges. The user's selection of controlpoints in the spectral and visual images may be saved to a repositorythat is used to provide a training correlation for personal and/orcustomized use. This approach may allow subjective best practices to beincorporated into the control point selection process.

In another variation according to aspects of the invention,software-based recognition of distinguishing features in the spectraland visual images may be used to select control points. The software maydetect at least one control point that corresponds to a distinguishingfeature in the visual or spectral images. For example, control points ina particular a cluster region may be selected in the spectral image. Thecluster pattern may be used to identify similar features in the visualimage. The control points may be used to digitally correlate the pixelsfrom the spectral image with the pixels from the visual image. Inanother aspect, the software may use morphological (e.g., shape)features in the images to select the control points. The morphologicalfeatures may come from the shape of the specimen, the shape of thespaces between the tissues, and/or the shape of stained regions withinthe tissue (e.g., as a result of staining the biological sample, forexample, with an IHC agent). Thus, any shape that may occur in thevisual image that also occurs in the spectral image may be used toselect the control points.

The features in both images may be aligned by translation, rotation, andscaling. Translation, rotation and scaling may also be automated orsemi-automated, for example, by developing mapping relationships ormodels after selecting the features selection. Such an automated processmay provide an approximation of mapping relationships that may then beresampled and transformed to optimize registration, for example.Resampling techniques include, but are not limited to nearest neighbor,linear, and cubic interpolation.

Once the control points are aligned, the pixels in the spectral imagehaving coordinates P₁ (x₁, y₁) may be aligned with the correspondingpixels in the visual image having coordinates P₂ (x₂, y₂). Thisalignment process may be applied to all or a selected portion of thepixels in the spectral and visual images. Once aligned, the pixels ineach of the spectral and visual images may be registered together. Bythis registration process, the pixels in each of the spectral image andvisual images may be digitally joined with the pixels in thecorresponding image. Since the method in accordance with aspects of theinvention allows the same biological sample to be testedspectroscopically and visually, the visual and spectral images may beregistered accurately.

An identification mark such as a numerical code, bar code, may be addedto the slide to verify that the correct specimen is being accessed. Thereference and identification marks may be recognized by a computer thatdisplays or otherwise stores the visual image of the biologicalspecimen. This computer may also contain software for use in imageregistration.

An example of image registration according to an aspect of the inventionis illustrated in FIGS. 14A-14C. FIG. 14A is a visual image of a smallcell lung cancer tissue sample, and FIG. 14B is spectral image of thesame tissue sample subjected to HCA. FIG. 14B contains spectral datafrom most of the upper right-hand section of the visual image of FIG.14A. When the visual image of FIG. 14A is registered with the spectralimage of FIG. 14B, the result is shown in FIG. 14C. As shown in FIG.14C, the circled sections containing spots and contours 1-4 that areeasily viewable in the spectral image of FIG. 14B correspond closely tothe spots and contours visible in the microscopic image of FIG. 14A.

Once the coordinates of the pixels in the spectral and visual images areregistered, they may be digitally stored together. The entire images ora portion of the images may be stored. For example, the diagnosticregions may be digitally stored instead of the images of the entiresample. This may significantly reduce data storage requirements.

A user who views a certain pixel region in either the spectral or visualimage may immediately access the corresponding pixel region in the otherimage. For example, a pathologist may select any area of the spectralimage, such as by clicking a mouse or with joystick control, and viewthe corresponding area of the visual image that is registered with thespectral image. FIG. 14D is an example of a graphical user interface(GUI) for the registered image of FIG. 14C according to aspects of theinvention. The GUI shown in FIG. 14D allows a pathologist to togglebetween the visual, spectral, and registered images and examine specificportions of interest.

In addition, as a pathologist moves or manipulates an image, he/she canalso access the corresponding portion of the other image to which it isregistered. For example, if a pathologist magnifies a specific portionof the spectral image, he/she may access the same portion in the visualimage at the same level of magnification.

Operational parameters of the visual microscope system, as well asmicroscope magnification, changes in magnification etc., may be alsostored in an instrument specific log file. The log file may be accessedat a later time to select annotation records and corresponding spectralpixels for training the algorithm. Thus, a pathologist may manipulatethe spectral image, and at a later time, the spectral image and thedigital image that is registered to it are both displayed at theappropriate magnification. This feature may be useful, for example,since it allows a user to save a manipulated registered image digitallyfor later viewing or for electronic transmittal for remote viewing.

Image registration may be used with a tissue section, a cell section,and/or any other biological sample having a known diagnosis, prognosis,and/or predictive use to extract training spectra during a training stepof a method in accordance with aspects of the invention. During thetraining step, a visual image of stained tissue may be registered withan unsupervised spectral image, such as from HCA. Image registration mayalso be used when making a diagnosis, prognosis, and/or predictive useon a tissue section. For example, a supervised spectral image of thetissue section may be registered with its corresponding visual image.Thus, a user may obtain a diagnosis, prognosis, and/or predictive usebased on any point in the registered images that has been selected.

Image registration according to aspects of the invention providesnumerous advantages over prior methods of analyzing biological samples.For example, it allows a pathologist to rely on a spectral image, whichreflects the highly sensitive biochemical content of a biologicalsample, when making analyzing biological material. As such, it providessignificantly greater accuracy in detecting small abnormalities,pre-cancerous, or cancerous cells, including micrometastates, than therelated art. Thus, the pathologist does not have to base his/heranalysis of a sample on his/her subjective observation of a visual imageof the biological sample. Thus, for example, the pathologist may simplystudy the spectral image and may easily refer to the relevant portion inthe registered visual image to verify his/her findings, as necessary.

In addition, the image registration method in accordance with aspects ofthe invention provides greater accuracy than the prior method of Bird(Bird et al., “Spectral detection of micro-metastates in lymph nodehisto-pathology”, J. Biophoton. 2, No. 1-2, 37-46 (2009)) because it isbased on correlation of digital data, i.e. the pixels in the spectraland visual images. Bird does not correlate any digital data from theimages, and instead relies merely on the skill of the user to visuallymatch spectral and visual images of adjacent tissue sections byphysically overlaying the images. Thus, the image registration method inaccordance with aspects of the invention provides more accurate andreproducible diagnoses with regard to abnormal or cancerous cells. Thismay be helpful, for example, in providing accurate diagnosis in theearly stages of disease, when indicia of abnormalities and cancer arehard to detect.

In an aspect, image registration may automatically occur between aspectral image and a visual image. For example, a computing system mayautomatically register a spectral image and a visual image based onfeatures of the images, as illustrated in FIGS. 5A-5C. In addition, acomputing system may automatically register a spectral image and avisual image based on coordinates that are independent of imagefeatures, as illustrated in FIGS. 4A and 4B.

Referring now to FIGS. 4A and 4B, illustrated is an example automatedmethod 400 for performing image registration based on coordinates thatare independent of image features, in accordance with an aspect of thepresent invention. For example, the method may be used when the spectralimage and visual image are captured using the same slide holder, such asa stage plate used with microscope stages. The slide holder may allowthe biological sample to be placed with spatial accuracy and precisionin each microscope.

The method may include receiving coordinate positions of a plurality ofreticles on a slide holder with a biological sample in a visualcollection apparatus 402. The visual collection apparatus may include,but is not limited to, a microscope that is capable of capturing animage of the biological sample. In addition, the slide holder mayinclude a plurality of reticles marking a coordinate location on theslide holder, as illustrated in FIG. 4C.

Referring now to FIG. 4C, illustrated is an example slide holder 426 inaccordance with an aspect of the present invention. Slide holder 426 mayinclude a slot 428 where the biological sample may be inserted.Biological samples may include, but are not limited to, cells andtissues. In addition, slide holder 426 may also include a plurality ofreticles 430, 432, and 434 marking a position on the slide holder 426.In another aspect, the plurality of reticles may be placed directly onthe slide, marking a position on the slide instead of the slide holder.Reticles 430, 432, and 434 may each have a coordinate location, e.g., an(x,y) coordinate. The coordinates from each of reticles 430, 432, and434 may define a coordinate system that may be used during dataacquisition of the biological sample. It should be noted that at leasttwo reticles may be used to determine the coordinate system. Forexample, when the coordinates of two reticles are used, two solutionsfor the coordinate systems may be generated by the computing system. Thecomputing system may select one of the two solutions generated basedupon the orientation of the biological sample. For example, thecomputing system may select the solution based upon the assumption thatthe biological sample is not turned upside down and/or flipped. Whenthree reticles are used to determine the coordinate system, a uniquesolution for the coordinate system may be generated by the computingsystem. Thus, as the number of reticles increase, the accuracy of thetransformation may also increase and/or improve

Referring back to FIG. 4A, the coordinate locations of each of thereticles on the slide holder may be received from the visual collectionapparatus. In an aspect, a computing system in communication with thevisual collection apparatus may receive the coordinate positions of theplurality of reticles on the slide holder. For example, the visualcollection apparatus (e.g., a microscope) may be programmed to locateeach of the reticles on the slide holder by moving the microscope and/orthe slide holder until the reticle comes into view and transmitting thecoordinate locations of the reticles to the computing system. In anotheraspect, a user may enter the coordinate locations of the reticles intothe computing system. For example, the user may move the microscopeand/or the slide holder until each reticle comes into view (and maybecome aligned with indicators, like crosshairs, within the microscope),and the user may enter the coordinates displayed on the microscope intothe computing system. Thus, it should be noted that a variety ofmechanisms, automated or otherwise, may be used to capture thecoordinate position of the reticles on the slide holder and send thecoordinate information to the computing system.

The method may also include receiving a visual image of the biologicalsample from the visual image collection apparatus 404. For example, thevisual image collection apparatus may transmit the visual image of thebiological sample captured by the visual image collection apparatus tothe computing system.

In addition, the method may include associating the coordinate positionsof the plurality of reticles on the slide holder with the visual image406 and storing the visual image coordinate positions and the visualimage 408. In an aspect, the computing system may associate thecoordinate positions of the reticles received with the visual imagereceived and store the visual image coordinate positions and the visualimage, for example, in a data repository. In an aspect, the computingsystem may associate the file that stores the received visual imagecoordinates with the file that stores the received visual image.

The method may further include receiving coordinate positions of theplurality of the reticles on the slide holder with the biological samplein a spectral image collection apparatus 410. It should be noted thatthe same slide holder with the biological sample that is used in thevisual image collection apparatus may also be used in the spectral imagecollection apparatus. The computing system may also be in communicationwith the spectral image collection apparatus and may receive thecoordinate positions of each of the plurality of the reticles on theslide holder directly from the spectral image collection apparatusand/or through a user of the spectral image collection apparatus. Forexample, the spectral collection apparatus may be programmed to locateeach of the reticles on the slide holder by moving the spectralcollection apparatus and/or the slide holder until the reticle comesinto view and sending the coordinate locations of the reticles to thecomputing system. A user may also enter the coordinate locations of thereticles into the computing system.

In addition, the method may include receiving a spectral image of thebiological sample from the spectral image collection apparatus 412. Thespectral image collection apparatus may transmit the captured spectralimage of the biological sample to the computing system.

The method may also include associating the coordinate positions of theplurality of reticles on the slide holder with the spectral image 414and storing the spectral image coordinates positions and the spectralimage 416. The computing system may associate the received spectralimage coordinates with the received spectral image. For example, thecomputing system may apply a label to the file storing the spectralimage coordinates associating the file to the spectral image. It shouldalso be noted that the spectral image coordinates may be stored in thesame file as the spectral image.

The method may further include aligning or otherwise associating thereceived visual image coordinates with the received spectral imagecoordinates 418. The computing system may automatically map the spectralimage coordinates to the visual image coordinates to create a commoncoordinate system between the visual image and the spectral image.

The method may additionally include generating a registered imagealigning the received spectral image and the received visual image basedupon the alignment of the visual image coordinates and the spectralimage coordinates 420. For example, the computing system may overlay thespectral image on the visual image using the alignment of the visualimage coordinates with the spectral image coordinates and automaticallygenerate a registered image. Thus, the computing system mayautomatically register the spectral image with the visual image by usingcoordinates that are independent of the features from the spectral imageand the visual image.

The method may optionally include storing the registered image 422. Thecomputing system may store the registered image in a data repository sothat a user of the computing system may access the registered imageand/or make changes to the registered image.

In addition, the method may optionally include optimizing the registeredimage 424. For example, the computing system may apply one or moreoptimizations to find the best rigid body transforms that will cause thevisual image coordinate points and the spectral image coordinate pointsto align or correspond. The computing system may use one or moreoptimizations to improve the accuracy of a registered image byattempting to further align the images.

One optimization may include minimizing the distance between thespectral image and the visual images in the overlaid images that aremapped in the same coordinate system. The distance may be a measure ofthe grayscale pixel-by-pixel errors summed over the whole image. Forexample, the optimization may include:

min(p)J=sum D(p,I1,I2)

D=I2(p)−I1

where p is the same scaled rigid body transformation used for the selectpoints or reticle-based registration, D(p, •, •) is the distance measure(which is applied pixel-by-pixel), and I2(p) is the I2 image transformedby p into the same space as I1. I1 and I2 images may be created byapplying a series of transformations to the spectral and visual imagesin order to get the images into the same grayscale space because thevisual pixel values may not directly compared to the HCA or spectralpixel values.

Another optimization may include minimizing the least squared errorbetween the pairs of points selected in the two images (e.g., the visualimage and the spectral image). In an aspect, the computing system mayperform an optimization to minimize the least squared error between thepairs of points selected in the two images (e.g., the visual image andthe spectral image). For example, the optimization may include:

${\min\limits_{p}J} = {{\sum\limits_{i,j}\left( {x_{i} - x_{j}} \right)^{2}} + \left( {y_{i} - y_{j}} \right)^{2}}$

where (x_(i), y_(i)) are selected reference points in the visual image,(x_(j), y_(j)) are the reference points from the spectral image afterthey are mapped to the visual image, and p=[u v λ θ] are theregistration parameters.

It should be noted that various optimization settings may be used by thecomputing system to provide optimization limits on the optimizationsbeing performed by the computing system. For example, optimizationlimits may include, but are not limited to, a maximum number of functionevaluations, convergence tolerances, and/or an upper and a lower boundon the transformation parameters. The upper and lower bounds may beadvantageous in preventing the optimization from venturing too faroutside of a desired solution.

Referring now to FIG. 5A, illustrated is an example method 500 forrefining image registration based on image features in accordance withan aspect of the present invention. The computing system may refine animage registration when the overlay of the visual image with thespectral image does not correspond well. For example, the overlay of theimages may display the features of the biological sample out ofalignment between the visual image and the spectral image. In an aspect,the computing system may automatically perform the method for refiningthe image registration to align the image features from the spectralimage with the visual image as precisely as possible. It should be notedthat the computing system may also switch between a first spectral imagelevel and a different spectral image level, for example, if the firstspectral image level does not contain sufficient useful information inthe spectral image when the registration occurs.

The method may include scaling the spectral image 502 and scaling thevisual image 504. Scaling may be performed so that the morphological(e.g., shape) features of interest are approximately the same size ineach image. Scaling may be based upon the ratios between the spectralimage and the visual images. For example, the visual image may have ahigher resolution than the spectral image, and therefore, the scalingmay include setting an upper and lower bound on the images to scale thehigher resolution image to a lower resolution image. Example scalingequations that the computing system may use include the following:

x _(H&E) =u+λ*(x _(HCA)*cos θ−y _(HCA)*sin θ)

y _(H&E) =v+λ*(y _(HCA)*sin θ+y _(HCA)*cos θ)

where (u,v) represents a translation, λ a scale factor and θ a rotation.The scaling may be applied to selected spectral image reference points(e.g., reticle coordinates and/or registration points selected by auser) and to map the selected spectral image reference points to thevisual image reference points (e.g., reticle coordinates and/orregistration points selected by the user).

In an aspect, the computing system may perform an optimization tominimize a registration distance function between all the points in theregistered images. For example, the optimization may include:

${\min\limits_{p}J} = {\sum\limits_{i,j}\left( {{I(p)}_{i,j} - T_{i,j}} \right)^{2}}$

where image T_(ij) is based on the visual image, I(p)_(ij) is based onthe spectral image further transformed by the scaled rigid bodytransform p=[u v λ θ] and interpolated into the same coordinate frame asT. The sum is over all pixels (i,j) in the images. While thisoptimization uses the grayscale distance function D=I(p)−T, it should benoted that other distance functions may be used in the optimization,such as the normalized gradient field. The images T and I areinterpolated, filtered and converted versions of the visual and spectralimages as required by the particular distance function being used.Different distance functions may require different conversions andfiltering of the visual and spectral images.

The method may optionally include normalizing the spectral image 505. Inan aspect, the computing system may normalize the spectral image toimprove the image features of the spectral images. For example, when thenormalized spectral image is compared with the visual image, thefeatures in the normalized spectral image may appear sharper and providea more accurate representation of the image features.

In an aspect, the computing system may apply a weighted normalization tothe spectral image. Using a weighted normalization on the spectral imagemay be beneficial, for example, because the infrared absorption spectrumof a cell or tissue pixel is dominated by the protein vibrations. Sinceproteins contribute over 60% of a cell's dry mass, whereas nucleic acids(DNA and RNA) contribute about 20% or less of a cell's dry mass.Therefore, the vibrations of proteins (observed predominantly in theamide I and II regions, between 1700 and 1500 cm⁻¹) may be much moreprominent in the spectra than the features of nucleic acids, which maybe observed mostly in the symmetric (ca. 1090 cm⁻¹) and antisymmetric(ca. 1230 cm⁻¹) phosphodiester stretching vibrations. Since changes innucleic acid vibrational bands are frequently observed with the onset ofcancerous disease, it may be advantageous to utilize normalizationprocedures that emphasize the low wavenumber spectral region of spectrain a data set. This approach may be advantageous, for example, whencarrying out hierarchical cluster analysis (HCA) for the initialpartition of the spectral data set.

In an aspect, the weighted normalization may include a ramp functionwith a value of 1 at the low wavenumber limit of the spectrum (typically778 cm⁻¹) and a value of 0 at the high wavenumber limit (typically 1800cm⁻¹) multiplied by the spectral vector after standard vectornormalization (e.g., “the ramp method”). The product of this functionmay include a weighted spectral vector in which the importance of theprotein region is suppressed.

In another aspect, the weighted normalization may include regionnormalization. In region normalization, the spectrum S is divided intotwo or more (for example 2 or 3) regions, such that the protein andnucleic acid spectral features fall within different regions (forexample: region 1 from 1800 to 1480 cm⁻¹, and region 2 from 1478 to 778cm⁻¹). The two (or more) regions may be vector normalized separately,adding more weight to the low intensity spectral regions. Although“region normalization” may cause a discontinuity (for example, between1478 and 1480 cm⁻¹) in the spectra, this method may result in betterdiscrimination of normal and cancerous regions, for example, as measuredby the number of clusters required in HCA for the discrimination ofnormal and cancerous regions.

The computing system may also perform the weighted normalization on thespectral image to improve the features of the spectral image to aid inthe refinement of the registered image.

The method may also include applying a threshold value to the spectralimage and the visual image 506 and generating a binary spectral imageand a binary visual image based on the applied threshold value 508. Thecomputing system may automatically select a threshold value to apply tothe spectral image and a threshold value to apply to the visual image.In addition, the computing system may receive the threshold values froma user of the computing system.

Referring now to FIG. 5B, illustrated is an example GUI interface thatallows a user to select a threshold value for the visual image 520 a andthe spectral image 520 b. For example, the user may use sliders to setthe threshold values. As the user moves the sliders, the tissues in bothimages may change color (e.g., black to white or white to black) and theshapes in or other aspects of the images may become more or less visibleor distinct, for example. The user may select a threshold value for eachof the spectral image and the visual image when the tissues in bothimages become the same color/shade (e.g., black or white) and commonshapes in each image may become visible without a lot of noise in theimage. It should be noted that the threshold values for the spectralimage and the visual image may be the same number and/or may be adifferent number.

Referring back to FIG. 5A, the computing system may receive the selectedthreshold values and generate a binary spectral image and a binaryvisual image based on the applied threshold values. For example, eachpixel with a number above the threshold value may be converted to white,while each pixel with a number below the threshold value may beconverted to black. The computing system may map all the pixels in thespectral image and the visual image into black and white using thethreshold values for each respective image. By generating a binary image(e.g., a black and white image), the interstitial spaces between thetissues may be highlighted, as well as the basic structure of thebiological sample, any morphological (e.g., shape) features in thebiological sample, and/or the shape of stained regions within the tissue(e.g., as a result of staining the biological sample, for example, withan IHC agent). Thus, any shape that may occur in the visual image thatalso occurs in the spectral image could be highlighted.

In addition, the computing system may display a difference imageillustrating the difference between the spectral binary image and one orboth of the visual binary image and the registered image. Referring nowto FIG. 5C, illustrated is an example GUI screen with a difference imageillustrated and a graph with points illustrating the progress inminimizing the error in the difference image. The difference imageprovides a visual indication of the accuracy of the fit of theregistered images. In an aspect, the difference image may be as black incolor as possible. If the difference image includes frequent whitespaces, this result may indicate the presence of error in the registeredimage (e.g., the overlay of the images illustrates image features out ofalignment). As illustrated in the graph, multiple iterations of thethreshold selection may occur before the difference image illustratesminimum error (e.g., a mostly black image) in the threshold selections.It should be noted that the iteration process may terminate if a maximumnumber of iterations is reached before the error is minimized in thedifference image.

Referring back to FIG. 5A, the computing system may continue to applyvarious threshold values until a minimum error value is reached for thebinary spectral image and the visual spectral image and/or until amaximum number of iterations are reached, whichever occurs first.

The method may also include applying a morphological closure to thebinary spectral image and the binary visual image 510. The morphologicalclosure may remove noise from the binary images by smoothing any tinydots that may appear in the binary images into white areas. For example,the computing system may apply the morphological closure to the binaryimages by adding a boundary to the images and converting the dots towhite into white areas and/or removing small black or white dots withinlarger white or black areas, respectively.

In addition, the method may include softening the edges in the binaryspectral image and the binary visual image 512. In an aspect, thecomputing system may apply a Gaussian filter to blur the ramps betweenthe black and white edges in the binary images. For example, thecomputing system may smooth across the edges to blur the edges and makethem softer, in order to improve convergence of the optimization.

The method may further include minimizing the grayscale differencebetween the binary spectral image and the binary visual image 514. Forexample, the computing system may apply one or more of the optimizationsdiscussed above in 502-512 to minimize the grayscale difference betweenthe spectral image and the binary image in order to obtain registrationparameters with a better fit. It should be noted that the optimizationprocess may repeat until the structures in the spectral image and thevisual image are aligned as close as possible in the registered image.

Training

A training set may optionally be developed, as set forth in step 305 inthe method provided in the flowchart of FIG. 3. According to aspects ofthe invention, a training set includes spectral data that is associatedwith specific diseases or conditions, among other things. Theassociation of diseases or conditions to spectral data in the trainingset may be based on a correlation of classical pathology to spectralpatterns based on morphological features normally found in pathologicalspecimens. The diseases and conditions may include, but are not limitedto, cellular abnormalities, inflammation, infections, pre-cancer, andcancer.

According to one aspect in accordance with the invention, in thetraining step, a training set may be developed by identifying a regionof a visual image containing a disease or condition, correlating theregion of the visual image to spectral data corresponding to the region,and storing the association between spectral data and the correspondingdisease or condition. The training set may then be archived in arepository, such as a database, and made available for use in machinelearning algorithms to provide a diagnostic algorithm with outputderived from the training set. The diagnostic algorithm may also bearchived in a repository, such as a database, for future use.

For example, a visual image of a tissue section may be registered with acorresponding unsupervised spectral image, such as one prepared by HCA.Then, a user may select a characteristic region of the visual image.This region may be classified and/or annotated by a user to specify adisease or condition. The spectral data underlying the characteristicregion in the corresponding registered unsupervised spectral image maybe classified and/or annotated with the disease or condition.

The spectral data that has been classified and/or annotated with adisease or condition provides a training set that may be used to train asupervised analysis method, such as an ANN. Such methods are alsodescribed, for example, in Lasch, Miljkovic Dupuy. The trainedsupervised analysis method may provide a diagnostic algorithm.

A disease or condition information may be based on algorithms that aresupplied with the instrument, algorithms trained by a user, or acombination of both. For example, an algorithm that is supplied with theinstrument may be enhanced by the user.

An advantage of the training step according to aspects of the inventionis that the registered images may be trained against the best available,consensus-based “gold standards”, which evaluate spectral data byreproducible and repeatable criteria. Thus, after appropriate instrumentvalidation and algorithm training, methods in accordance with aspects ofthe invention may produce similar results worldwide, rather than relyingon visually-assigned criteria such as normal, atypical, low gradeneoplasia, high grade neoplasia, and cancer. The results for each cellmay be represented by an appropriately scaled numeric index or theresults overall as a probability of a classification match. Thus,methods in accordance with aspects of the invention may have thenecessary sensitivity and specificity for the detection of variousbiological structures, and diagnosis of disease.

The diagnostic limitation of a training set may be limited by the extentto which the spectral data are classified and/or annotated with diseasesor conditions. As indicated above, this training set may be augmented bythe user's own interest and expertise. For example, a user may preferone stain over another, such as one or many IHC stains over an H&Estain. In addition, an algorithm may be trained to recognize a specificcondition, such as breast cancer metastases in axillary lymph nodes, forexample. The algorithm may be trained to indicate normal vs. abnormaltissue types or binary outputs, such as adenocarcenoma vs.not-adenocarcenoma only, and not to classify the different normal tissuetypes encountered, such as capsule, B- and T-lymphocytes. The regions ofa particular tissue type, or states of disease, obtained by SHP, may berendered as “digital stains” superimposed on real-time microscopicdisplays of the tissue sections.

Diagnosis, Prognosis, Predictive, Thernostic

Once the spectral and visual images have been registered, they may beused make a medical diagnosis, as outlined in step 306 in the flowchartof FIG. 3. The diagnosis may include a disease or condition including,but not limited to, cellular abnormalities, inflammation, infections,pre-cancer, cancer, and gross anatomical features. In a method accordingto aspects of the invention, spectral data from a spectral image of abiological specimen of unknown disease or condition that has beenregistered with its visual image may be input to a trained diagnosticalgorithm, as described above. Based on similarities to the training setthat was used to prepare the diagnostic algorithm, the spectral data ofthe biological specimen may be correlated to a disease or condition. Thedisease or condition may be output as a diagnosis.

For example, spectral data and a visual image may be acquired from abiological specimen of unknown disease or condition. The spectral datamay be analyzed by an unsupervised method, such as HCA, which may thenbe used along with spatial reference data to prepare an unsupervisedspectral image. This unsupervised spectral image may be registered withthe visual image, as discussed above. The spectral data that has beenanalyzed by an unsupervised method may then be input to a trainedsupervised algorithm. For example, the trained supervised algorithm maybe an ANN, as described in the training step above. The output from thetrained supervised algorithm may be spectral data that contains one ormore labels that correspond to classifications and/or annotations of adisease or condition based on the training set.

To extract a diagnosis based on the labels, the labeled spectral datamay used to prepare a supervised spectral image that may be registeredwith the visual image and/or the unsupervised spectral image of thebiological specimen. For example, when the supervised spectral image isregistered with the visual image and/or the unsupervised spectral image,through a GUI, a user may select a point of interest in the visual imageor the unsupervised spectral image and be provided with a disease orcondition corresponding to the label at that point in the supervisedspectral image. As an alternative, a user may request a software programto search the registered image for a particular disease or condition,and the software may highlight the sections in any of the visual,unsupervised spectral, and supervised spectral images that are labeledwith the particular disease or condition. This advantageously allows auser to obtain a diagnosis in real-time, and also allows the user view avisual image, which he/she is familiar with, while accessing highlysensitive spectroscopically obtained data.

The diagnosis may include a binary output, such as an “is/is not” typeoutput, that indicates the presence or lack of a disease or condition.In addition, the diagnosis may include, but is not limited to anadjunctive report, such as a probability of a match to a disease orcondition, an index, or a relative composition ratio.

In accordance with aspects of the method of the invention, grossarchitectural features of a tissue section may be analyzed via spectralpatterns to distinguish gross anatomical features that are notnecessarily related to disease. Such procedures, known as global digitalstaining (GDS), may use a combination of supervised and unsupervisedmultivariate methods. GDS may be used to analyze anatomical featuresincluding, but not limited to, glandular and squamous epithelium,endothelium, connective tissue, bone, and fatty tissue.

In GDS, a supervised diagnostic algorithm may be constructed from atraining dataset that includes multiple samples of a given disease fromdifferent patients. Each individual tissue section from a patient may beanalyzed as described above, using spectral image data acquisition,pre-processing of the resulting dataset, and analysis by an unsupervisedalgorithm, such as HCA. The HCA images may be registered withcorresponding stained tissue, and may be annotated by a pathologist.This annotation step, indicated in FIGS. 15A-C, allows the extraction ofspectra corresponding to typical manifestation of tissue types ordisease stages and states, or other desired features. The resultingtypical spectra, along with their annotated medical diagnosis, maysubsequently be used to train a supervised algorithm, such as an ANN,that is specifically suited to detect the features it was trained torecognize.

According to the GDS method, the sample may be stained using classicalstains or immuno-histochemical agents. When the pathologist receives thestained sample and inspects it using a computerized imaging microscope,the spectral results may be available to the computer controlling thevisual microscope. The pathologist may select any tissue spot on thesample and receive a spectroscopy-based diagnosis. This diagnosis mayoverlay a grayscale or pseudo-color image onto the visual image thatoutlines all regions that have the same spectral diagnosticclassification.

FIG. 15A is a visual microscopic image of H&E-stained lymph node tissuesection. FIG. 15B shows a typical example of global discrimination ofgross anatomical features, such as capsule and interior of lymph node.FIG. 15B is a global digital staining image of section shown in FIG.15A, distinguishing capsule and interior of lymph node.

Areas of these gross anatomical features, which are registered with thecorresponding visual image, may be selected for analysis based on moresophisticated criteria in the spectral pattern dataset. This next levelof diagnosis may be based on a diagnostic marker digital staining (DMDS)database, which may be solely based on SHP results, for example, or maycontain spectral information collected using immuno-histochemical (IHC)results. For example, a section of epithelial tissue may be selected toanalyze for the presence of spectral patterns indicative of abnormalityand/or cancer, using a more diagnostic database to scan the selectedarea. An example of this approach is shown schematically in FIG. 15C,which utilizes the full discriminatory power of SHP and yields detailsof tissue features in the lymph node interior (such as cancer,lymphocytes, etc.), as may be available only after immune-histochemicalstaining in classical histopathology. FIG. 15C is a DMDS image ofsection shown in FIG. 15A, distinguishing capsule, metastatic breastcancer, histiocytes, activated B-lymphocytes and T-lymphocytes.

The relationship between GDS and DMDS is shown by the horizontalprogression marked in dark blue and purple, respectively, in theschematic of FIG. 16. Both GDS and DMDS are based on spectral data, butmay include other information, such as IHC data. The actual diagnosismay also be carried out by the same or a similarly trained diagnosticalgorithm, such as a hANN. Such a hANN may first analyze a tissuesection for gross anatomical features detecting large variance in thedataset of patterns collected for the tissue (the dark blue track).Subsequent “diagnostic element” analysis may be carried out by the hANNusing a subset of spectral information, shown in the purple track. Amulti-layer algorithm in binary form may be implemented, for example.Both GDS and DMDS may use different database subsections, shown as GrossTissue Database and Diagnostic Tissue Database in FIG. 16, to arrive atthe respective diagnoses, and their results may be superimposed on thestained image after suitable image registration.

According to an example method in accordance with aspects of theinvention, a pathologist may provide certain inputs to ensure that anaccurate diagnosis is achieved. For example, the pathologist mayvisually check the quality of the stained image. In addition, thepathologist may perform selective interrogation to change themagnification or field of view of the sample.

The method according to aspects of the invention may be performed by apathologist viewing the biological specimen and performing the imageregistration. Alternatively, since the registered image contains digitaldata that may be transmitted electronically, the method may be performedremotely.

Methods may be demonstrated by the following non-limiting examples.

Example 2 Lymph Node Section

FIG. 17 shows a visual image of an H&E-stained axillary lymph nodesection measuring 1 mm×1 mm, containing a breast cancer micrometastasisin the upper left quadrant. FIG. 17B is a SHP-based digitally stainedregion of breast cancer micrometastasis. By selecting, for example, byclicking using a cursor controlled mouse, in the general area of themicrometastasis, a region that was identified by SHP to be cancerous ishighlighted in red as shown in FIG. 17B. FIG. 17C is a SHP-baseddigitally stained region occupied by B-lymphocyes. By pointing towardthe lower right corner, regions occupied by B-lymphocyte are marked inlight blue, as shown in FIG. 17C. FIG. 17D is a SHP-based digitallystained region that shows regions occupied by histocytes, which areidentified by the arrow.

Since the SHP-based digital stain is based on a trained and validatedrepository or database containing spectra and diagnoses, the digitalstain rendered is directly relatable to a diagnostic category, such as“metastatic breast cancer,” in the case of FIG. 17B. The system may befirst used as a complementary or auxiliary tool by a pathologist,although the diagnostic analysis may be carried out by SHP. As anadjunctive tool, the output may be a match probability and not a binaryreport, for example. FIG. 18 shows the detection of individual and smallclusters of cancer cells with SHP.

Example 3 Fine Needle Aspirate Sample of Lung Section

Sample sections were cut from formalin fixed paraffin embedded cellblocks that were prepared from fine needles aspirates of suspiciouslegions located in the lung. Cell blocks were selected based on thecriteria that previous histological analysis had identified anadenocarcinoma, small cell carcinoma (SCC) or squamous cell carcinoma ofthe lung. Specimens were cut by use of a microtome to provide athickness of about 5 μm and subsequently mounted onto low-e microscopeslides (Kevley Technologies, Ohio, USA). Sections were thendeparaffinized using standard protocols. Subsequent to spectroscopicdata collection, the tissue sections were hematoxylin and eosin (H&E)stained to enable morphological interpretations by a histopathologist.

A Perkin Elmer Spectrum 1/Spotlight 400 Imaging Spectrometer (PerkinElmer Corp, Shelton, Conn., USA) was employed in this study. Infraredmicro-spectral images were recorded from 1 mm×1 mm tissue areas intransflection (transmission/reflection) mode, with a pixel resolution of6.25 μm×6.25 μm, a spectral resolution of 4 cm⁻¹, and the co-addition of8 interferograms, before Norton-Beer apodization (see, e.g., Naylor, etal. J Opt. Soc. Am., A24:3644-3648 (2007)) and Fourier transformation.An appropriate background spectrum was collected outside the sample areato ratio against the single beam spectra. The resulting ratioed spectrawere then converted to absorbance. Each 1 mm×1 mm infrared imagecontains 160×160, or 25,600 spectra.

Initially, raw infrared micro-spectral data sets were imported into andprocessed using software written in Matlab (version R2009a, Mathworks,Natick, Mass., USA). A spectral quality test was performed to remove allspectra that were recorded from areas where no tissue existed, ordisplayed poor signal to noise. All spectra that pass the test were thenbaseline off-set normalized (subtraction of the minimal absorbanceintensity across the entire spectral vector), converted to secondderivative (Savitzy-Golay algorithm (see, e.g., Savitzky, et al. Anal.Chem., 36:1627 (1964)), 13 smoothing points), cut to only includeintensity values recorded in the 1350 cm⁻¹-900 cm⁻¹ spectral region, andfinally vector normalized.

Processed data sets were imported into a software system and HCAperformed using the Euclidean distance to define spectral similarity,and Ward's algorithm (see, e.g., Ward, J Am. Stat. Assoc., 58:236(1963)) for clustering. Pseudo-color cluster images that describe pixelcluster membership, were then assembled and compared directly with H&Eimages captured from the same sample. HCA images of between 2 and 15clusters, which describe different clustering structures, were assembledby cutting the calculated HCA dendrogram at different levels. Thesecluster images were then provided to collaborating pathologists whoconfirmed the clustering structure that best replicated themorphological interpretations they made upon the H&E-stained tissue.

Infrared spectra contaminated by underlying base line shifts,unaccounted signal intensity variations, peak position shifts, orgeneral features not arising from or obeying LambertBeer law werecorrected by a sub-space model version of EMSC for Mie scattering andreflection contributions to the recorded spectra (see B. Bird, M.Miljković and M. Diem, “Two step resonant Mie scattering correction ofinfrared micro-spectral data: human lymph node tissue”, J. Biophotonics,3 (8-9) 597-608 (2010)). Initially, 1000 recorded spectra for eachcancer type were pooled into separate data sets from the infrared imagespresented in FIG. 19A-19F.

These data sets were then searched for spectra with minimal scatteringcontributions, a mean for each cancer type was calculated to increasesignal to noise, and KK transforms were calculated for each cell type,as shown in FIG. 19A and FIG. 19B. FIG. 19A shows raw spectral data setscomprising cellular spectra recorded from lung adenocarcinoma, smallcell carcinoma, and squamous cell carcinoma cells. FIG. 19B showscorrected spectral data sets comprising cellular spectra recorded fromlung adenocarcinoma, small cell carcinoma, and squamous cell carcinomacells, respectively. FIG. 19C shows standard spectra for lungadenocarcinoma, small cell carcinoma, and squamous cell carcinoma.

A sub space model for Mie scattering contributions was constructed bycalculating 340 Mie scattering curves that describe a nuclei sphereradius range of 6 μm-40 μm, and a refractive index range of 1.1-1.5,using the Van de Hulst approximation formulae (see, e.g., Brussard, etal., Rev. Mod. Phys., 34:507 (1962)). The first 10 principal componentsthat describe over 95% of the variance composed in these scatteringcurves, were then used in a addition to the KK transforms for eachcancer type, as interferences in a 1 step EMSC correction of data sets.The EMSC calculation took approximately 1 sec per 1000 spectra. FIG. 19Dshows KK transformed spectra calculated from spectra in FIG. 19C. FIG.19E shows PCA scores plots of the multi class data set before EMSCcorrection. FIG. 19F shows PCA scores plots of the multi class data setafter EMSC correction. The analysis was performed on the vectornormalized 1800 cm⁻¹-900 cm⁻¹ spectral region.

FIG. 20A shows mean absorbance spectra of lung adenocarcinoma, smallcell carcinoma, and squamous carcinoma, respectively. These werecalculated from 1000 scatter corrected cellular spectra of each celltype. FIG. 20B shows second derivative spectra of absorbance spectradisplayed in FIG. 20A. In general, adenocarcinoma and squamous cellcarcinoma have similar spectral profiles in the low wavenumber region ofthe spectrum. However, the squamous cell carcinoma displays asubstantially low wavenumber shoulder for the amide I band, which hasbeen observed for spectral data recorded from squamous cell carcinoma inthe oral cavity (Papamarkakis, et al. (2010), Lab. Invest., 90:589-598).The small cell carcinoma displays very strong symmetric andanti-symmetric phosphate bands that are shifted slightly to higherwavenumber, indicating a strong contribution of phospholipids to theobserved spectra.

Since the majority of sample area is composed of blood andnon-diagnostic material, the data was pre-processed to only includediagnostic material and correct for scattering contributions. Inaddition, HCA was used to create a binary mask and finally classify thedata. This result is shown in FIGS. 21A-21C. FIG. 21A shows 4 stitchedmicroscopic R&E-stained images of 1 mm×1 mm tissue areas comprisingadenocarcinoma, small cell carcinoma, and squamous cell carcinoma cells,respectively. FIG. 21B is a binary mask image constructed by performanceof a rapid reduced RCA analysis upon the 1350 cm⁻¹-900 cm⁻¹ spectralregion of the 4 stitched raw infrared images recorded from the tissueareas shown in FIG. 21A. The regions of diagnostic cellular material andblood cells are shown. FIG. 21C is a 6-cluster RCA image of the scattercorrected spectral data recorded from regions of diagnostic cellularmaterial. The analysis was performed on the 1800 cm⁻¹-900 cm⁻¹ spectralregion. The regions of squamous cell carcinoma, adenicarcinoma, smallcell carcinoma, and diverse desmoplastic tissue response are shown.Alternatively, these processes can be replaced with a supervisedalgorithm, such as an ANN.

The results presented in the Examples above show that the analysis ofraw measured spectral data enables the differentiation of SCC andnon-small cell carcinoma (NSCC). After the raw measured spectra arecorrected for scattering contributions, adenocarinoma and squamous cellcarcinoma according to methods in accordance with aspects of theinvention, however, the two subtypes of NSCC, are clearlydifferentiated. Thus, these Examples provide strong evidence that thisspectral imaging method may be used to identify and correctly classifythe three main types of lung cancer.

FIG. 22 shows various features of an example computer system 100 for usein conjunction with methods in accordance with aspects of invention,including, but not limited to image registration and training. As shownin FIG. 22, the computer system 100 may be used by a requestor 101 via aterminal 102, such as a personal computer (PC), minicomputer, mainframecomputer, microcomputer, telephone device, personal digital assistant(PDA), or other device having a processor and input capability. Theserver module may comprise, for example, a PC, minicomputer, mainframecomputer, microcomputer, or other device having a processor and arepository for data or that is capable of accessing a repository ofdata. The server module 106 may be associated, for example, with anaccessible repository of disease based data for use in diagnosis.

Information relating to a diagnosis, for example, via a network, 110,such as the Internet, for example, may be transmitted between theanalyst 101 and the server module 106. Communications may be made, forexample, via couplings 111, 113, such as wired, wireless, or fiberopticlinks.

Aspects of the invention may be implemented using hardware, software ora combination thereof and may be implemented in one or more computersystems or other processing systems. In one variation, aspects of theinvention are directed toward one or more computer systems capable ofcarrying out the functionality described herein. An example of such acomputer system 200 is shown in FIG. 23.

Computer system 200 includes one or more processors, such as processor204. The processor 204 is connected to a communication infrastructure206 (e.g., a communications bus, cross-over bar, or network). Varioussoftware aspects are described in terms of this exemplary computersystem. After reading this description, it will become apparent to aperson skilled in the relevant art(s) how to implement the aspects ofinvention using other computer systems and/or architectures.

Computer system 200 can include a display interface 202 that forwardsgraphics, text, and other data from the communication infrastructure 206(or from a frame buffer not shown) for display on the display unit 230.Computer system 200 also includes a main memory 208, preferably randomaccess memory (RAM), and may also include a secondary memory 210. Thesecondary memory 210 may include, for example, a hard disk drive 212and/or a removable storage drive 214, representing a floppy disk drive,a magnetic tape drive, an optical disk drive, etc. The removable storagedrive 214 reads from and/or writes to a removable storage unit 218 in awell-known manner. Removable storage unit 218, represents a floppy disk,magnetic tape, optical disk, etc., which is read by and written toremovable storage drive 214. As will be appreciated, the removablestorage unit 218 includes a computer usable storage medium having storedtherein computer software and/or data.

In alternative variations, secondary memory 210 may include othersimilar devices for allowing computer programs or other instructions tobe loaded into computer system 200. Such devices may include, forexample, a removable storage unit 222 and an interface 220. Examples ofsuch may include a program cartridge and cartridge interface (such asthat found in video game devices), a removable memory chip (such as anerasable programmable read only memory (EPROM), or programmable readonly memory (PROM)) and associated socket, and other removable storageunits 222 and interfaces 220, which allow software and data to betransferred from the removable storage unit 222 to computer system 200.

Computer system 200 may also include a communications interface 224.Communications interface 224 allows software and data to be transferredbetween computer system 200 and external devices. Examples ofcommunications interface 224 may include a modem, a network interface(such as an Ethernet card), a communications port, a Personal ComputerMemory Card International Association (PCMCIA) slot and card, etc.Software and data transferred via communications interface 224 are inthe form of signals 228, which may be electronic, electromagnetic,optical or other signals capable of being received by communicationsinterface 224. These signals 228 are provided to communicationsinterface 224 via a communications path (e.g., channel) 226. This path226 carries signals 228 and may be implemented using wire or cable,fiber optics, a telephone line, a cellular link, a radio frequency (RF)link and/or other communications channels. In this document, the terms“computer program medium” and “computer usable medium” are used to refergenerally to media such as a removable storage drive 214, a hard diskinstalled in hard disk drive 212, and signals 228. These computerprogram products provide software to the computer system 200. Aspects ofthe invention are directed to such computer program products.

Computer programs (also referred to as computer control logic) arestored in main memory 208 and/or secondary memory 210. Computer programsmay also be received via communications interface 224. Such computerprograms, when executed, enable the computer system 200 to perform thefeatures in accordance with aspects of the invention, as discussedherein. In particular, the computer programs, when executed, enable theprocessor 204 to perform such features. Accordingly, such computerprograms represent controllers of the computer system 200.

In a variation where aspects of the invention are implemented usingsoftware, the software may be stored in a computer program product andloaded into computer system 200 using removable storage drive 214, harddrive 212, or communications interface 224. The control logic(software), when executed by the processor 204, causes the processor 204to perform the functions as described herein. In another variation,aspects of the invention are implemented primarily in hardware using,for example, hardware components, such as application specificintegrated circuits (ASICs). Implementation of the hardware statemachine so as to perform the functions described herein will be apparentto persons skilled in the relevant art(s).

In yet another variation, aspects of the invention are implemented usinga combination of both hardware and software.

What is claimed is:
 1. A method for registering a visual image and aspectral image of a biological sample, comprising: receiving a first setof coordinate positions of a plurality of reticles on a slide holderwith the biological sample; receiving a visual image of the biologicalsample; receiving a second set of coordinate positions of the pluralityof reticles on the slide holder with the biological sample; receiving aspectral image of the biological sample; aligning the first set ofcoordinates and the second set of coordinates; and generating aregistered image with the spectral image and the visual image based uponthe alignment of the first set of coordinates and the second set ofcoordinates.
 2. The method of claim 1, wherein generating the registeredimage occurs automatically based upon the alignment of the first set ofimage coordinate and the second set of image coordinates.
 3. The methodof claim 1, wherein the first set of coordinates and the second set ofcoordinates create a common coordinate system for the registered image.4. The method of claim 1, wherein the first set of coordinates areindependent from image features within the visual image.
 5. The methodof claim 1, wherein the second set of coordinates are independent fromimage features within the spectral image.
 6. The method of claim 1,further comprising: performing optimization of the registered image. 7.A system for registering a visual image and a spectral image of abiological sample, comprising: a processor; a user interface functioningvia the processor; and a repository accessible by the processor; whereina first set of coordinate positions of a plurality of reticles on aslide holder with the biological sample is received; wherein a visualimage of the biological sample is received; wherein a second set ofcoordinate positions of the plurality of reticles on the slide holderwith the biological sample is received; wherein a spectral image of thebiological sample is received; wherein the first set of coordinates isaligned with the second set of coordinates; and wherein a registeredimage is generated with the spectral image and the visual image basedupon the alignment of the first set of coordinates and the second set ofcoordinates.
 8. A computer program product comprising a computer usablemedium having control logic stored therein for causing a computer toregister a visual image and a spectral image of a biological sample, thecontrol logic comprising: a set of codes for causing a computer toreceive a first set of coordinate positions of a plurality of reticleson a slide holder with the biological sample; a set of codes for causingthe computer to receive a visual image of the biological sample; a setof codes for causing the computer to receive a second set of coordinatepositions of the plurality of reticles on the slide holder with thebiological sample; a set of codes for causing the computer to receive aspectral image of the biological sample; a set of codes for causing thecomputer to align the first set of coordinates and the second set ofcoordinates; and a set of codes for causing the computer to generate aregistered image with the spectral image and the visual image based uponthe alignment of the first set of coordinates and the second set ofcoordinates.
 9. A method for image registration refinement, comprising:receiving a spectral image of a biological sample and a visual image ofthe biological sample; applying a first threshold value to the spectralimage and a second threshold value to the visual image; and generating abinary spectral image based on the first threshold value; generating abinary visual image based on the second threshold value; aligningbiological features of the spectral image with biological features ofthe visual image using the binary spectral image and the binary visualimage; and refining an image registration of the spectral image and thevisual image based upon the alignment of the biological features of thespectral image and the biological features of the visual image.
 10. Themethod of claim 9, wherein aligning the features of the spectral imageand the features of the visual image improve the image registration ofthe spectral image and the visual image.
 11. The method of claim 9,further comprising: applying a weighted normalization to the spectralimage to improve image features of the spectral image.
 12. The method ofclaim 9, further comprising: applying a morphological closure to thebinary spectral image and the binary visual image.
 13. The method ofclaim 9, further comprising: softening the edges in the binary spectralimage and the binary visual image.
 14. The method of claim 9, furthercomprising: minimizing the grayscale difference between the binaryspectral image and the binary visual image.
 15. The method of claim 9,further comprising: scaling the visual image; and scaling the spectralimage.
 16. A system for image registration refinement, comprising: aprocessor; a user interface functioning via the processor; and arepository accessible by the processor; wherein a spectral image of abiological sample and a visual image of the biological sample isreceived; wherein a first threshold value is applied to the spectralimage and a second threshold value is applied to the visual image;wherein a binary spectral image is generated based on the firstthreshold value; wherein a binary visual image is generated based on thesecond threshold value; wherein biological features of the spectralimage are aligned with biological features of the visual image using thebinary spectral image and the binary visual image; and wherein an imageregistration of the spectral image and the visual image is refined basedupon the alignment of the biological features of the spectral image andthe biological features of the visual image.
 17. A computer programproduct comprising a computer usable medium having control logic storedtherein for causing a computer to refine a registered image, the controllogic comprising: a set of codes for causing a computer to receive aspectral image of a biological sample and a visual image of thebiological sample; a set of codes for causing the computer to apply afirst threshold value to the spectral image and a second threshold valueto the visual image; and a set of codes for causing the computer togenerate a binary spectral image based on the first threshold value; aset of codes for causing the computer to generate a binary visual imagebased on the second threshold value; a set of codes for causing thecomputer to align biological features of the spectral image withbiological features of the visual image using the binary spectral imageand the binary visual image; and a set of codes for causing the computerto refine an image registration of the spectral image and the visualimage based upon the alignment of the biological features of thespectral image and the biological features of the visual image.